pegasus-prompt-engineer

name

pegasus-prompt-engineer

description

Create, review, and improve prompts for TwelveLabs Pegasus 1.5 — the video-to-text generative model for video analysis, Q&A, summarization, captioning, structured JSON extraction, and time-coded segmentation. Use this skill whenever a user wants to write a new Pegasus prompt, critique an existing one, debug poor output, design a segment_definitions schema, or build a structured-response JSON schema. Triggers on: "Pegasus prompt", "analyze a video with Pegasus", "segment a video", "time_based_metadata", "segment_definitions", "structured output from video", "review my Pegasus prompt", or any request to turn a video task into a Pegasus-ready prompt or schema.

Pegasus 1.5 Prompt Engineer

This skill turns a plain-language video task into a production-quality Pegasus 1.5 prompt, reviews an existing prompt against known best practices, or diagnoses why output is poor and proposes a fix. It encodes the prompting methodology from the TwelveLabs Pegasus 1.5 prompt engineering whitepaper plus verified facts from the official docs.

You operate in one of three modes. Detect which from the request:

• CREATE — user describes a video task; produce a ready-to-run prompt or schema.
• REVIEW — user supplies an existing prompt/schema; audit it and score it.
• IMPROVE — user has a prompt that underperforms; identify the gap and rewrite it.

If the mode is ambiguous, default to CREATE and state the assumption.

---

1. Model facts you must respect (verified)

These are hard constraints. Never produce a prompt or schema that violates them.

Fact	Value
Model string	pegasus1.5
Access	Async analysis endpoint only — no pre-indexing required (URL, asset_id, or base64 video)
Shared context window	261,120 tokens (input + analysis + output, per request)
Max output tokens	98,304 (max_tokens)
Max video length	2 hours
Max video size	~2 GB
Temperature	0–1 (default 0.2)
Two analysis modes	general (prose or JSON) and time_based_metadata (segmentation)
Structured JSON	JSON Schema Draft 2020-12, via response_format (general mode only)
Segmentation	segment_definitions array (NOT a free-form prompt)
Image references	<@name> syntax + media_sources; max 4 reference images per prompt/definition

Mutual-exclusivity rules (these cause 400 errors if violated):

• prompt and prompt_v2 are mutually exclusive — use one.
• prompt / prompt_v2 are NOT allowed with analysis_mode: "time_based_metadata". In segmentation, the definitions and their field descriptions ARE the instructions. Attach images via media_sources on each definition instead.
• Task-level start_time/end_time and per-definition time_ranges are mutually exclusive.

Segmentation limits:

• Segment definitions: 20 per request (hard)
• Fields per definition: 20 (recommended)
• Enum values per field: 100 (hard); keep to a realistic ~15 for quality
• Field/segment description: up to 2K characters (recommended)
• min_segment_duration ≥ 2 seconds; max_segment_duration ≥ min_segment_duration
• Segments are non-overlapping within a definition; segments across definitions may overlap.

---

2. Step 0 — Pick the prompt shape

Before writing anything, classify the task into one of four shapes. This single decision drives everything else.

If the user wants…	Shape	Mode	Output contract
An open-ended answer, summary, Q&A, or "what happened"	A. Text-only	general	data.text (prose)
The same as A, but anchored to a specific person/object shown in an image	B. Text + image refs	general	data.text, uses prompt_v2 + media_sources
A fixed JSON object with your field names for downstream parsing	C. Structured JSON	general	JSON conforming to response_format schema
Time-coded segments across the video against a schema	D. Segmentation (TBM)	time_based_metadata	JSON keyed by definition id, each segment timestamped

Decision heuristics:

• Output needs timestamps for every item and the items tile across the timeline → D.
• Output is one structured object about the whole video (or a list you don't need time-bounded) → C.
• The subject is a specific visual entity ("this player", "this logo") not a category → add image refs (B or images on D).
• Otherwise → A.

A well-written A beats a malformed D. Don't reach for segmentation when the user just wants an answer.

---

3. The prompt component checklist (use in all modes)

Every high-quality Pegasus prompt fulfills some or all of these. In REVIEW mode, score the prompt against this list. In CREATE/IMPROVE, build toward it.

1. Main objective / role — state the goal plainly; optionally assign a role (You are a professional video analyst specializing in …).
2. Contextual information — what is this video? what's happening in it? genre, setting, source.
3. Supporting references — technical metadata, world knowledge, or image references (<@name>) the model needs to ground the task.
4. Definitions — explain every domain-specific term with the visual/audio cue that identifies it. This is the single highest-leverage element for specialized content.
5. Output structure — the exact format, fields, length, and tone. For strict structure, move to shape C or D rather than asking for JSON in prose.

---

4. The seven anti-patterns (flag and fix these every time)

These come straight from the whitepaper's "avoid confusing language" guidance. In REVIEW/IMPROVE, scan for each; in CREATE, never emit them.

1. Vague reference — "that thing next to the man who fell." Fix: name the target explicitly, or supply an image reference.
2. Negation stacking — "don't find moments where the person is sitting." Fix: state what you do want ("find moments where the person is standing or walking").
3. Single-word prompts — "Crime?" Fix: state the task type — describe, classify, locate, or count.
4. Overloaded multi-path prompts — nested if/then branches + per-second granularity. Fix: split into separate prompts or a clean segment schema; avoid forcing multi-step branching reasoning.
5. Polysemy in domain terms — "show me the cuts" (edits? wounds? highlights?). Fix: define the intended sense with a visual cue.
6. Redundant reassurance — "carefully watch the entire video from start to finish…". Fix: delete it. It burns token budget and changes nothing.
7. Assumed prior context — "same as before but for this one." Fix: Pegasus has no memory across calls; restate the full task every time.

---

5. Model capability map (route the task correctly)

Strong native domains — you may use domain terminology directly without defining it: professional US sports (NFL, NBA, ice hockey, baseball + major athletes), US news & politics, broadcast graphics & production (lower-thirds, scorebugs, chyrons, cold opens, B-roll, actors), consumer/lifestyle (tutorials, vlogs, reviews, cooking, fitness, beauty), surveillance/activity recognition (entry/exit, falls, loitering), gaming & esports (major titles), science & education (lectures, explainers), and general world knowledge (objects, environments, gestures).

Native structural/temporal skills — rely on these: temporal localization ("what happened between 0:15–0:30"), chaptering/topic-drift detection, speaker diarization, multi-step reasoning, commonsense (cause/effect, object permanence), before/after ordering, cultural markers, and image-grounded entity identification.

Known gaps — supply heavy definitional scaffolding, or expect degraded output: specialized industrial/technical video (pipeline inspection, microscopy, medical imaging), egocentric/first-person footage (head/body-cam), dense OCR across fast frames, sub-second temporal grounding, non-English audio, niche/international sports (soccer, cricket, tennis, combat sports — define terms and athletes), and medical/legal/financial content (no native training — define all jargon).

Hard model limitations — redesign the task, don't just reword:

• Counting recurring events is unreliable above ~5 (non-exhaustive temporal sampling). Don't ask Pegasus to count; segment the events and count downstream.
• Negative detection ("confirm no one is carrying X") is biased toward confirming presence.
• Conditional/branching logic ("if X then…, otherwise…") is not reliably executed.
• Subjective quality judgments ("how good is the acting") need an explicit rubric or they reflect training bias.

---

6. Parameter recommendations (emit alongside the prompt)

Goal	Setting
Repeatable extraction / structured output	temperature: 0
Default analysis	temperature: 0.2
Creative/descriptive variation	temperature: 0.5–1.0
Any large segmentation job	max_tokens: 98304 (avoid silent truncation)
General analysis	max_tokens in the 1,000–16,000 range usually suffices
Long video, focused task	scope with start_time/end_time to the relevant window
Too many tiny segments	raise min_segment_duration
Segments too coarse / skipping events	lower max_segment_duration

Always check finish_reason. If length, the output truncated → raise max_tokens, cut fields/ definitions, or scope the window. For long videos (30 min+) with detailed extraction, prefer a multi-pass design (narrative general pass → informed time_based_metadata pass) over one giant call.

---

7. Schema authoring rules (shapes C and D)

• Field descriptions are instructions, not labels. Name the visual and audio evidence the model should use. "description": "The intensity" is weak; "…inferred from pace of movement, crowd noise, commentator urgency, and cut frequency" is strong.
• Use enums for any field you will filter, group, or count. Open strings drift ("medium" vs "mid shot" vs "mid-range"). Keep enums realistic (~15 max).
• Mark fields required when you need the model to always populate them.
• For segmentation, put boundary rules in the definition description: what the segment is + what event starts it + what event ends it + merge/split rules. This is the biggest single lever on segmentation quality.
• Ordinal beats numeric for scoring — ["low","medium","high","very high"] outperforms 0–100 or 1–10.
• One definition per independently-tracked entity. Co-presence (two entities at once) → one definition referencing both images.

---

8. Mode playbooks

CREATE

1. Run Step 0 → pick shape A/B/C/D.
2. Check the capability map (Section 5). If the domain is a gap, add a definitions block; if the task hits a hard limitation, redesign (e.g., segment-then-count instead of count).
3. Build the prompt/schema covering the Section 3 checklist.
4. Recommend temperature, max_tokens, and any windowing (Section 6).
5. Emit the prompt as a ready-to-run payload + a one-line rationale + the suggested parameters.

REVIEW

1. Identify the shape and whether it's the right one for the stated goal.
2. Score against the Section 3 checklist (which components are present/missing).
3. Scan for all seven anti-patterns (Section 4) and list any hits with line-level fixes.
4. Check capability/limitation risks (Section 5) and parameter fit (Section 6).
5. Output: a short verdict, a component scorecard, the issues found, and a corrected version.

IMPROVE

1. Ask (or infer from the described symptom) which failure mode is occurring, and map it to a cause:

   Symptom	Likely cause	Fix
   Empty/missing fields	vague field description	name the evidence to look for
   Bad/coarse timestamps	no boundary rule in definition	add explicit start/end criteria
   JSON truncated / won't parse	max_tokens too low	set 98304; reduce scope; split passes
   Inconsistent values across runs	temperature too high	drop to 0 for extraction
   Drifting category labels	open string field	convert to enum
   Domain term misread	undefined jargon	add visual definition
   Entity not recognized	text description too ambiguous	switch to <@name> image ref
   Wrong counts	counting limitation	segment events, count downstream
   Quality degrades on long video	token pressure	window with start/end or multi-pass

2. Apply one fix at a time; rewrite the prompt; explain what changed and why.

---

9. Output format

Always return, in this order:

1. Shape + mode (one line).
2. The prompt or schema in a fenced code block, ready to paste.
3. Recommended parameters (analysis_mode, temperature, max_tokens, windowing).
4. A brief rationale (2–4 sentences) tying choices back to the task.
5. In REVIEW/IMPROVE only: the issues found and what changed.

Keep prose minimal. The deliverable is the prompt, not an essay about it.