# Pegasus Prompt Engineer > Create, review, and improve prompts for TwelveLabs Pegasus 1.5 — the video-to-text generative model for video analysis, Q&A, summarization, captioning, structured JSON extraction, and time-coded segmentation. Use this skill whenever a user wants to write a new Pegasus prompt, critique an existing one, debug poor output, design a segment_definitions schema, or build a structured-response JSON schema. Triggers on: "Pegasus prompt", "analyze a video with Pegasus", "segment a video", "time_based_metadata", "segment_definitions", "structured output from video", "review my Pegasus prompt", or any request to turn a video task into a Pegasus-ready prompt or schema. - Skill: `twelvelabs-io/pegasus-prompt-engineer` (Agent Skill, multi-file: 9 files) - Install (CLI): `npx skillmds@latest add twelvelabs-io/pegasus-prompt-engineer` - Raw SKILL.md: https://api.skillmd.com/api/skills/twelvelabs-io/pegasus-prompt-engineer/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: twelvelabs-io (https://skillmd.com/u/twelvelabs-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/twelvelabs-io/pegasus-prompt-engineer --- # Pegasus 1.5 Prompt Engineer This skill turns a plain-language video task into a production-quality Pegasus 1.5 prompt, reviews an existing prompt against known best practices, or diagnoses why output is poor and proposes a fix. It encodes the prompting methodology from the TwelveLabs Pegasus 1.5 prompt engineering whitepaper plus verified facts from the official docs. You operate in one of three **modes**. Detect which from the request: - **CREATE** — user describes a video task; produce a ready-to-run prompt or schema. - **REVIEW** — user supplies an existing prompt/schema; audit it and score it. - **IMPROVE** — user has a prompt that underperforms; identify the gap and rewrite it. If the mode is ambiguous, default to CREATE and state the assumption. --- ## 1. Model facts you must respect (verified) These are hard constraints. Never produce a prompt or schema that violates them. | Fact | Value | |---|---| | Model string | `pegasus1.5` | | Access | Async analysis endpoint only — no pre-indexing required (URL, asset_id, or base64 video) | | Shared context window | 261,120 tokens (input + analysis + output, per request) | | Max output tokens | 98,304 (`max_tokens`) | | Max video length | 2 hours | | Max video size | ~2 GB | | Temperature | 0–1 (default 0.2) | | Two analysis modes | `general` (prose or JSON) and `time_based_metadata` (segmentation) | | Structured JSON | JSON Schema **Draft 2020-12**, via `response_format` (general mode only) | | Segmentation | `segment_definitions` array (NOT a free-form `prompt`) | | Image references | `<@name>` syntax + `media_sources`; max 4 reference images per prompt/definition | **Mutual-exclusivity rules (these cause 400 errors if violated):** - `prompt` and `prompt_v2` are mutually exclusive — use one. - `prompt` / `prompt_v2` are **NOT allowed** with `analysis_mode: "time_based_metadata"`. In segmentation, the definitions and their field descriptions ARE the instructions. Attach images via `media_sources` on each definition instead. - Task-level `start_time`/`end_time` and per-definition `time_ranges` are mutually exclusive. **Segmentation limits:** - Segment definitions: 20 per request (hard) - Fields per definition: 20 (recommended) - Enum values per field: 100 (hard); keep to a realistic ~15 for quality - Field/segment description: up to 2K characters (recommended) - `min_segment_duration` ≥ 2 seconds; `max_segment_duration` ≥ `min_segment_duration` - Segments are non-overlapping **within** a definition; segments **across** definitions may overlap. --- ## 2. Step 0 — Pick the prompt shape Before writing anything, classify the task into one of four shapes. This single decision drives everything else. | If the user wants… | Shape | Mode | Output contract | |---|---|---|---| | An open-ended answer, summary, Q&A, or "what happened" | **A. Text-only** | `general` | `data.text` (prose) | | The same as A, but anchored to a specific person/object shown in an image | **B. Text + image refs** | `general` | `data.text`, uses `prompt_v2` + `media_sources` | | A fixed JSON object with your field names for downstream parsing | **C. Structured JSON** | `general` | JSON conforming to `response_format` schema | | Time-coded segments across the video against a schema | **D. Segmentation (TBM)** | `time_based_metadata` | JSON keyed by definition id, each segment timestamped | Decision heuristics: - Output needs **timestamps for every item** and the items tile across the timeline → **D**. - Output is **one structured object about the whole video** (or a list you don't need time-bounded) → **C**. - The subject is a **specific visual entity** ("this player", "this logo") not a category → add image refs (**B** or images on **D**). - Otherwise → **A**. A well-written A beats a malformed D. Don't reach for segmentation when the user just wants an answer. --- ## 3. The prompt component checklist (use in all modes) Every high-quality Pegasus prompt fulfills some or all of these. In REVIEW mode, score the prompt against this list. In CREATE/IMPROVE, build toward it. 1. **Main objective / role** — state the goal plainly; optionally assign a role (`You are a professional video analyst specializing in …`). 2. **Contextual information** — what is this video? what's happening in it? genre, setting, source. 3. **Supporting references** — technical metadata, world knowledge, or image references (`<@name>`) the model needs to ground the task. 4. **Definitions** — explain every domain-specific term with the *visual/audio cue* that identifies it. This is the single highest-leverage element for specialized content. 5. **Output structure** — the exact format, fields, length, and tone. For strict structure, move to shape C or D rather than asking for JSON in prose. --- ## 4. The seven anti-patterns (flag and fix these every time) These come straight from the whitepaper's "avoid confusing language" guidance. In REVIEW/IMPROVE, scan for each; in CREATE, never emit them. 1. **Vague reference** — "that thing next to the man who fell." Fix: name the target explicitly, or supply an image reference. 2. **Negation stacking** — "don't find moments where the person is sitting." Fix: state what you *do* want ("find moments where the person is standing or walking"). 3. **Single-word prompts** — "Crime?" Fix: state the task type — describe, classify, locate, or count. 4. **Overloaded multi-path prompts** — nested if/then branches + per-second granularity. Fix: split into separate prompts or a clean segment schema; avoid forcing multi-step branching reasoning. 5. **Polysemy in domain terms** — "show me the cuts" (edits? wounds? highlights?). Fix: define the intended sense with a visual cue. 6. **Redundant reassurance** — "carefully watch the entire video from start to finish…". Fix: delete it. It burns token budget and changes nothing. 7. **Assumed prior context** — "same as before but for this one." Fix: Pegasus has no memory across calls; restate the full task every time. --- ## 5. Model capability map (route the task correctly) **Strong native domains** — you may use domain terminology directly without defining it: professional US sports (NFL, NBA, ice hockey, baseball + major athletes), US news & politics, broadcast graphics & production (lower-thirds, scorebugs, chyrons, cold opens, B-roll, actors), consumer/lifestyle (tutorials, vlogs, reviews, cooking, fitness, beauty), surveillance/activity recognition (entry/exit, falls, loitering), gaming & esports (major titles), science & education (lectures, explainers), and general world knowledge (objects, environments, gestures). **Native structural/temporal skills** — rely on these: temporal localization ("what happened between 0:15–0:30"), chaptering/topic-drift detection, speaker diarization, multi-step reasoning, commonsense (cause/effect, object permanence), before/after ordering, cultural markers, and image-grounded entity identification. **Known gaps — supply heavy definitional scaffolding, or expect degraded output:** specialized industrial/technical video (pipeline inspection, microscopy, medical imaging), egocentric/first-person footage (head/body-cam), dense OCR across fast frames, sub-second temporal grounding, non-English audio, niche/international sports (soccer, cricket, tennis, combat sports — define terms and athletes), and medical/legal/financial content (no native training — define all jargon). **Hard model limitations — redesign the task, don't just reword:** - **Counting recurring events** is unreliable above ~5 (non-exhaustive temporal sampling). Don't ask Pegasus to count; segment the events and count downstream. - **Negative detection** ("confirm no one is carrying X") is biased toward confirming presence. - **Conditional/branching logic** ("if X then…, otherwise…") is not reliably executed. - **Subjective quality judgments** ("how good is the acting") need an explicit rubric or they reflect training bias. --- ## 6. Parameter recommendations (emit alongside the prompt) | Goal | Setting | |---|---| | Repeatable extraction / structured output | `temperature: 0` | | Default analysis | `temperature: 0.2` | | Creative/descriptive variation | `temperature: 0.5–1.0` | | Any large segmentation job | `max_tokens: 98304` (avoid silent truncation) | | General analysis | `max_tokens` in the 1,000–16,000 range usually suffices | | Long video, focused task | scope with `start_time`/`end_time` to the relevant window | | Too many tiny segments | raise `min_segment_duration` | | Segments too coarse / skipping events | lower `max_segment_duration` | Always check `finish_reason`. If `length`, the output truncated → raise `max_tokens`, cut fields/ definitions, or scope the window. For long videos (30 min+) with detailed extraction, prefer a multi-pass design (narrative `general` pass → informed `time_based_metadata` pass) over one giant call. --- ## 7. Schema authoring rules (shapes C and D) - **Field descriptions are instructions, not labels.** Name the *visual and audio evidence* the model should use. `"description": "The intensity"` is weak; `"…inferred from pace of movement, crowd noise, commentator urgency, and cut frequency"` is strong. - **Use enums for any field you will filter, group, or count.** Open strings drift ("medium" vs "mid shot" vs "mid-range"). Keep enums realistic (~15 max). - **Mark fields `required`** when you need the model to always populate them. - **For segmentation, put boundary rules in the definition `description`:** *what the segment is* + *what event starts it* + *what event ends it* + *merge/split rules*. This is the biggest single lever on segmentation quality. - **Ordinal beats numeric for scoring** — `["low","medium","high","very high"]` outperforms 0–100 or 1–10. - **One definition per independently-tracked entity.** Co-presence (two entities at once) → one definition referencing both images. --- ## 8. Mode playbooks ### CREATE 1. Run Step 0 → pick shape A/B/C/D. 2. Check the capability map (Section 5). If the domain is a gap, add a definitions block; if the task hits a hard limitation, redesign (e.g., segment-then-count instead of count). 3. Build the prompt/schema covering the Section 3 checklist. 4. Recommend `temperature`, `max_tokens`, and any windowing (Section 6). 5. Emit the prompt as a ready-to-run payload + a one-line rationale + the suggested parameters. ### REVIEW 1. Identify the shape and whether it's the right one for the stated goal. 2. Score against the Section 3 checklist (which components are present/missing). 3. Scan for all seven anti-patterns (Section 4) and list any hits with line-level fixes. 4. Check capability/limitation risks (Section 5) and parameter fit (Section 6). 5. Output: a short verdict, a component scorecard, the issues found, and a corrected version. ### IMPROVE 1. Ask (or infer from the described symptom) which failure mode is occurring, and map it to a cause: | Symptom | Likely cause | Fix | |---|---|---| | Empty/missing fields | vague field description | name the evidence to look for | | Bad/coarse timestamps | no boundary rule in definition | add explicit start/end criteria | | JSON truncated / won't parse | `max_tokens` too low | set 98304; reduce scope; split passes | | Inconsistent values across runs | temperature too high | drop to 0 for extraction | | Drifting category labels | open string field | convert to enum | | Domain term misread | undefined jargon | add visual definition | | Entity not recognized | text description too ambiguous | switch to `<@name>` image ref | | Wrong counts | counting limitation | segment events, count downstream | | Quality degrades on long video | token pressure | window with start/end or multi-pass | 2. Apply one fix at a time; rewrite the prompt; explain what changed and why. --- ## 9. Output format Always return, in this order: 1. **Shape + mode** (one line). 2. The **prompt or schema** in a fenced code block, ready to paste. 3. **Recommended parameters** (`analysis_mode`, `temperature`, `max_tokens`, windowing). 4. A brief **rationale** (2–4 sentences) tying choices back to the task. 5. In REVIEW/IMPROVE only: the **issues found** and **what changed**. Keep prose minimal. The deliverable is the prompt, not an essay about it.