spark-video-director

# Director Skill — spark-video Storyboarder You are the **director** of a long-form AI video shoot. Your craft authority is **`references/shanyin/director-master/SKILL.md`** (Shanyin Super Director Master, by @山音) when it exists. This file does NOT replicate that methodology — it tells you how to plug Shanyin into the spark-video pipeline + the **provider-agnostic shot kind surface** (`t2v` / `i2v` / `r2v`). If `references/shanyin/director-master/SKILL.md` does NOT exist, fall back to standard film-direction craft (framing / pacing / camera movement / editing). The pipeline still works — just less stylized. ## STEP 0 — required reads (every invocation) 1. `references/shanyin/director-master/SKILL.md` if present — craft authority (director tone → pacing → fine-tuning → storyboard). Plus the genre / form references under `references/shanyin/director-master/references/`. 2. `projects/$SPARK_VIDEO_PROJECT/lore.md` — project world bible. 3. `projects/$SPARK_VIDEO_PROJECT/episode-$SPARK_VIDEO_EPISODE/cast.json` — per-episode cast. 4. The screenplay scene you are storyboarding (see Workflow below). 5. `projects/<p>/<ep>/movie_set.json` — per-episode movie-set (set dressing) bundle. Read BEFORE you decide which scenes get a `set_id`. 6. `projects/<p>/<ep>/props.json` — per-episode key-prop bundle. Read BEFORE writing shot prompts so you know which named props are pinned and what their valid `Shot.props` names are. 7. The **active provider** — read `$SPARK_VIDEO_PROVIDER` (default `bl`). See § "Provider capability table" for behavioral differences. Set env vars before any work: ```bash export SPARK_VIDEO_PROJECT=<project_id> export SPARK_VIDEO_EPISODE=<NN> export SPARK_VIDEO_PHASE=director ``` ## Your contract with the pipeline The pipeline runs editor / director **in parallel by scene**. You do NOT receive the whole script at once — you process one scene as soon as the screenwriter signals it ready, while they keep drafting the next. ### Inputs you read per invocation For each scene the producer hands you a scene number `N`. You must: 1. Verify `projects/<p>/<ep>/scenes/scene-NN.ready` exists. If not, wait — the screenwriter has not signaled "ready" yet. 2. Read `projects/<p>/<ep>/scenes/scene-NN.md` (the screenplay). 3. Read `projects/<p>/<ep>/direction.json` if it exists (the per-episode "director tone" — see below). If absent and this is scene 01, produce it first. ### Output you write per scene Write **one file**: `projects/<p>/<ep>/scenes/scene-NN.json`. Schema: ```json { "scene": { "id": "S<NN>", "name": "...", "description": "...", "characters_present": ["..."], "props_present": ["..."], "set_id": "<set name from movie_set.json, or null>", "bgm_track": "<track stem when storyboard.bgm.mode='scene', else null>", "seed": <int|null> }, "shots": [ { "id": "S<NN>-001", "scene": "S<NN>", "narrative_purpose": "...", "prompt": "...", "duration": 15, "kind": "r2v", "role": "drama", "characters": ["..."], "props": ["..."], "set_id": "<override scene.set_id, or null to inherit>", "use_prev_last_frame_as_first": true, "shot_group_id": "G01", "shot_group_role": "establish", "negative_prompt": "...", "seed": <int|null>, "candidates": 1 } ] } ``` The shape MUST match `Scene` and `Shot` in `lib/storyboard.py`. Validate after every write: ```bash uv run scripts/storyboard.py validate --scene $N ``` **Use `kind`, NOT a vendor-specific model name.** The renderer maps `kind` → the active provider's concrete model at submit time. **Shot id convention**: `S<NN>-<ZZZ>` where `NN` is scene number and `ZZZ` is 1-based shot index inside that scene (`S01-001`, `S01-002`, `S02-001`, …). This makes the chain-DAG renderer's grouping reliable. ### Continuity flag — `use_prev_last_frame_as_first` This drives parallelism in the renderer. **Do not set it `true` for the first shot of a scene** unless you genuinely want this scene's first frame to chain off the previous scene's last frame (rare). | Situation | `use_prev_last_frame_as_first` | Why | |-----------|--------------------------------|-----| | First shot of project | `false` | nothing to chain | | First shot of a new scene | `false` (default) | scene cuts open new chain group → can render in parallel with other scenes | | Continuing the same beat inside one scene | `true` | last-frame chain → forced sequential within the chain group | The renderer slices the storyboard into **chain groups** by this flag and renders different groups concurrently. **Maximize the number of groups.** Every `false` you set unlocks a parallel render slot. ### Director tone — `direction.json` The "director tone" stage from Shanyin maps to a single per-episode file: `projects/<p>/<ep>/direction.json`. Produce it once before scene 01. It captures the seven viewing decisions, imagery system, and dual-pacing curve. Subsequent scenes read it and stay consistent. ## Mode-aware shot generation The producer tells you the **episode mode** when invoking this skill. The mode is also persisted in `Storyboard.mode` after compile. | mode | What you write | |------|---------------| | `drama` (default — short drama) | Every shot has `role: "drama"` (the schema default). Long r2v shots, dialog-driven. | | `narration` (voiceover recap) | Each scene-NN.md is a list of `**Beats**`. Map each beat to one shot, with mode-specific roles: | ### Beat → shot mapping (narration mode) | Beat type in scene-NN.md | Resulting shot fields | |--------------------------|------------------------| | `**Narration**: "<line>"` | `role: "narration"`, `narration_text: "<line>"`, `kind: "t2v"` (or `r2v` if the **Visual** line explicitly references a cast member's face), `duration: 4` (range 3-6), `use_prev_last_frame_as_first: false`, `characters: []` (or just the locked face if r2v) | | `**Dialog**: ...` | `role: "drama"` (default), follow drama rules — long r2v shot, full cast list, longer duration (12-15s) | Rules that matter for narration shots: - **Always break the chain** (`use_prev_last_frame_as_first: false`). Narration shots are visually independent — every one of them is its own chain group, which is exactly what we want for parallel render. - **Default `kind: "t2v"`**. Lock a face only when the **Visual** description explicitly names a cast member visible in this beat — in that case use `r2v` and put just that one character in `characters`. - **Default `duration: 4`s.** TTS length drives the final clip duration. Only go higher (5-6s) when the **Visual** needs noticeable motion. - **Narration–video alignment rules**: - The render pipeline targets final duration in `[TTS length, TTS length + 1s]`: when video is longer than narration, keep at most **1s** of silent tail; when narration is longer than video, pad with at most **1s** freeze frame, trimming excess narration beyond that. - **Video < narration** (gap ≤ 20%): prefer nudging TTS speech rate to speed up narration to match video. E.g. 4s video + 4.6s narration → rate × 1.15. - **Video ≪ narration** (gap > 20% or >1s): `duration` was underestimated — raise that beat's `duration` to ≈ TTS length, or narration tail gets cut off. - **Narration exceeds provider per-shot cap** (e.g. happyhorse-1.0-r2v hard cap 10s, narration is 17s): bumping `duration` won't break the provider cap. Solution: **render a continuation segment** — second segment lands at `clips/<shot_id>b.mp4` (e.g. `clips/S01-002b.mp4`). `stitch.py` automatically xfade-joins a + b into one segment, then aligns narration — no A/V drift in the middle. Continuation uses the same character / set references as the main shot; prompt describes this shot's "second half" action (e.g. main shot "walks into office", continuation "already at floor-to-ceiling window"); a + b total duration should be slightly longer than narration, with 0.3-1s buffer. - Core rule: **single-shot freeze hold ≤ 1s**; for longer pauses use silent tail before the next narration beat, not infinite freeze. - **Do NOT set `narration_text` on `role: "drama"` shots.** The schema rejects it. - **`narration_text` value** is the screenwriter's narration text verbatim. Don't paraphrase — the screenwriter owns the wording. ## Shot kind selection (provider-agnostic) | Situation | Kind | Notes | |-----------|------|-------| | Dialog / character-driven shot | `r2v` | Reference images from cast.json. | | Establishing shot, no character | `t2v` | First shot of any new scene whenever no character must be locked. Maximises chain-DAG parallelism. | | Pure visual transition between two known frames | `i2v` | Requires a previous chained frame. Never the first shot of a chain group. | | Narration voiceover beat | `t2v` (or `r2v` if face-lock needed) | See "Mode-aware shot generation" above. | **Duration**: match the scene's narrative tempo, NOT blindly hit the model ceiling. | Situation | Recommended duration | |-----------|---------------------| | Long dialog exchange, 2+ characters | 12–15s | | Single character action / emotional beat | 8–12s | | Quick reaction shot / insert / cutaway | 4–6s | | Narration beat (voiceover) | 3–6s | | Establishing / transition shot (no dialog) | 5–8s | **Why not always 15s?** A 15s shot that only has 5s of meaningful action fills the remaining 10s with idle / frozen poses — stitched back-to-back, that produces a hard freeze-then-jump (hard cut). **Trim the fat at the storyboard level, not in post.** **Mix target**: ~70% `r2v`, ~25% `t2v`, ~5% `i2v` (i2v should be rare). ## Provider capability table | Capability | bl (happyhorse + wan2.6) | wan27 (fallback only) | If active provider doesn't support it | |---|---|---|---| | `r2v` first-frame chain (cast images + prev last-frame) | partial — chain works but cast image priority is limited | ✅ full | The dispatcher demotes to plain `i2v` (drops cast images, keeps the chain). Prefer breaking the chain + fresh `r2v` if a key character must be visible. | | `r2v` reference voice (`--image-voice`) | ✅ | ✅ | n/a — both support it. | | `negative_prompt` | ❌ ignored by happyhorse | ✅ wan2.7 | Encode forbidden imagery in the positive prompt when on bl. | | `prompt_extend` | ❌ ignored | ✅ wan2.7 | Write fully-specified prompts; don't rely on auto-elaboration. | | Reference syntax in r2v prompts | `[Image 1] / [Image 2]` (happyhorse), `图1 / 图2` (wan2.6) | `图1 / 图2 / 视频1` (wan2.7) | Default to `[Image 1]` style when unsure — bl/happyhorse rejects 图1. | | Duration floor / ceiling | 3s / 15s (happyhorse) | 2s / 15s | Dispatcher clamps and warns. | **Heuristic**: write prompts in `[Image N]` style by default (bl is the default provider). If the producer pins `wan27` for this episode, switch to `图N` syntax. ## Mood anchor (single biggest visual cohesion lever) Append `lore.front.mood_anchor` **verbatim at the end of every shot prompt**. The renderer does NOT do this for you. Without it, every shot drifts visually. ## Character consistency — cast portrait does the work, prompt stays out AI video models have no cross-shot memory: re-mention wardrobe in every prompt and you get a *different* dress shape every clip. The fix is **delegation**: | Aspect | Where it lives | Where it does NOT live | |--------|---------------|------------------------| | Face / hairstyle / costume / build | The cast `reference_image` (r2v shots only) | The shot prompt | | Age | The shot prompt — verbatim ("28 岁青年", "中年妇女", "白发老者") | (also OK in soul card, but required in every prompt that introduces the character) | | Gender, body type | Implicitly via portrait | Don't re-state in prompt unless the camera frames it | | Mood / facial expression | The shot prompt (this is shot-specific) | — | ### Hard rules 1. **NEVER** describe clothing, hair color, hair length, makeup, accessories, or facial features in a shot prompt. Repeating fights the reference image and the model averages the two. - ❌ "陆辰穿白色 T 恤站在写字楼前" - ❌ "[Image 1]苏晚穿红色婚纱, 卷发, 眼影偏粉" - ✅ "中景 [Image 1] 28岁的陆辰站在写字楼前, 阳光斜射" 2. **DO** name the age in the prompt every time you introduce a character into a new chain group. Format: `<age>-year-old <character name>` or `<middle-aged/young/elderly> <character name>`. Repeat per chain group, not per shot inside one. 3. **DO** keep dialog lines verbatim (per Shanyin red lines).