| name | music-video |
| description | Generate a complete music video from a YAML spec — song via suno-mcp, per-scene audio-reactive video via comfyui LTX-2.3 ia2v, final concatenation with the clean song audio overlaid. Use when the user asks for a music video, long-form video synced to music, multi-scene video with a soundtrack, or anything that chains multiple ia2v clips into one output. Triggers on: "music video", "video to a song", "make a song and a video", "clip to this song", "visualizer", "lyric video".
|
| metadata | {"openclaw":{"emoji":"🎬","requires":{"skills":["comfyui","suno-mcp"]}}} |
Music Video Skill
Author a YAML spec → run one command → get a full music video.
The loop
Nine-step workflow with two quality-check gates where the pipeline stops
and waits for a human (or reviewing agent) to confirm before the next
expensive phase. A bad song variant or a broken base anchor will ruin every
downstream scene — don't skip the gates unless you're sure.
-
init <slug> — creates ~/.openclaw/workspace/<slug>/ with a
skeleton song.yaml. Use --theme "<text>" to inject context for the
lyrics/style fields. Example:
music_video.py init new-beginnings --theme "fresh start"
-
Fill song.yaml — title, style (producer-style brief — see
~/.openclaw/skills/suno-mcp/references/style-guide.md), and lyrics
([Verse]/[Chorus]/[Bridge] tags, one line per phrase, no mid-line
punctuation — see
~/.openclaw/skills/suno-mcp/references/lyrics-guide.md). Leave scenes
empty for now.
-
song <spec> — generates spec.suno.runs * 2 suno variants
(song.mp3, song_v2.mp3, song_v3.mp3, ...). Restart-safe: re-runs
only top up missing variants.
-
QUALITY GATE 1 — pick the best variant. Listen to every song*.mp3,
pick the one you want, back up or delete song.mp3, then rename your
chosen variant to song.mp3. Keep the others as song_vN.mp3 — they
become parallel final_vN.mp4 renders during assembly.
-
MANDATORY — transcribe the chosen song for word-level timestamps.
You cannot eyeball scene start_sec/duration_sec. Whisper STT against
the actual audio is the source of truth. Run:
comfy_graph.py stt --audio song.mp3 --prefix <slug> \
--output-dir <project-dir> --language English
Use the segment SRT cues to set start_sec/duration_sec so each
scene starts on a vocal phrase boundary and ends just before the
next one. The orchestrator will slice the song audio per scene at
exactly these boundaries, so misalignment shows up as words cut in
half across scene cuts.
If you have separated stems (e.g. from Suno's stem-export tool, or
comfy_graph.py stems on the full mix), transcribe the relevant stems
individually and merge. Lead vocals, backing vocals, and the full
mix can each surface lines the others miss:
- Lead-vocal stem alone may drop hooks/answer-phrases that live only
in the backing-vocals layer (e.g. a refrain "Stable altitude"
answered only by the backing vocal).
- Whisper on the full mix can hallucinate text where backing-vocal
bleed sounds like a vocal phrase.
Run STT on each stem with a different
--prefix, diff the segment
SRTs, and reconcile timing conflicts in favour of the cleanest stem
for each line. Don't trust a single pass.
Defensive note: if stt fails with HTTP 413, your ComfyUI server
(or the proxy in front of it) has a body-size limit on /upload/image
below your audio file's size. Either fix the proxy or re-encode the
audio for the STT pass only:
ffmpeg -i song.mp3 -c:a libmp3lame -b:a 24k -ar 16000 -ac 1 song_stt.mp3
— that re-encoded copy is fine for transcription but do NOT use it as
the ia2v audio source (LTX's audio VAE wants the full-quality input).
If Suno gave you a stem pack (zip of .wav + .mid per stem),
unpack it first so the MIDI is available to the bar-grid analyser:
unpack_suno_pack.py <Title>.zip --out <project-dir>
Skipping this step costs you the drum tempo map and the
per-stem MIDI phrase boundaries — both crucial for the next step.
5b. OPTIONAL but recommended — combine MIDI bar grid + vocal phrases
+ Whisper STT into a ranked scene-boundary proposal. This catches
musically-wrong cuts that the segment SRT alone won't:
```bash
analyze_midi.py <project-dir> # report tempo / bars
analyze_song.py <project-dir> \
--whisper-srt <project>/<prefix>_segments.srt \
--scenes <N> --min-spacing 4
```
The output is a table of `[start, dur, lyric]` per scene, snapped to
bar boundaries where possible, that you can drop into `scenes:`.
Mostly: the song's BPM dictates a bar length (e.g. 123 BPM → 1.944s);
every scene end that doesn't land within ~100ms of a bar will feel
musically off in playback. Use this output as the starting point even
if you adjust some boundaries by hand.
6. Add the scene list to song.yaml — each scene: label, start_sec,
duration_sec, prompt, image (@anchor | @last | path). Optionally
give a scene an anchor: block to pre-render a flux2 key frame for it.
Lift start_sec/duration_sec straight from analyze_song.py's
output (step 5b) when you have a MIDI, or from <prefix>_segments.srt
(step 5) otherwise. Total duration must equal the song length (probe
with ffmpeg -i song.mp3 if unknown).
Validate every boundary against <prefix>_words.srt before
rendering. Segment-level SRT marks line endings, but vocalists
routinely sustain final phonemes 0.3-1s past the line's nominal end —
if your scene ends inside that sustain, LTX cuts the operator's mouth
in the middle of "do" or "say" and the next scene's overlay audio
carries the lingering vowel. Open _words.srt, find the line you're
cutting on, and ensure your start_sec + duration_sec lands on the
END of the last word in the line (or in a clear silent gap), not the
"end" reported by the segment SRT. A 30-line audit script:
python3 -c "import re; ..." over _words.srt flagging boundaries
that fall inside word spans is faster than reviewing the final video.
7. anchors <spec> — flux2 renders the top-level anchor_image (from
anchor_prompt, or fallback title + style) plus any per-scene anchors.
Idempotent — skips files already on disk.
8. QUALITY GATE 2 — review the anchors. Open every PNG under
<project>/ and <project>/scenes/. If anything is wrong (wrong face,
wrong mood, bad composition), DELETE that PNG and re-run anchors <spec>
— tweak the scene's anchor.prompt or top-level anchor_prompt first.
A bad base anchor propagates into every LTX scene that chains off it.
9. scenes <spec> — renders every scene (LTX ia2v with the song slice +
resolved image). Restart-safe. Iterate one at a time with
scene N <spec>.
10. assemble <spec> — concat all scene MP4s, overlay the clean suno
audio → final.mp4 (plus final_vN.mp4 for each extra song variant).
Running the whole thing
music_video.py all <spec> runs steps 3→10 in order. It STOPS automatically
at gates 1 and 2 (controlled by gate_confirm_song / gate_confirm_anchors
in the YAML, default both true). To run fully unattended, either flip both
flags to false in song.yaml or pass --no-gate:
music_video.py all <spec>
music_video.py all --no-gate <spec>
Restart-safe throughout: anything already on disk is skipped. Iterate
individual scenes with scene N song.yaml.
Day-1 decisions (lock these before step 7)
A few choices propagate into every downstream prompt and every render. Flipping
them mid-pipeline forces you to throw away anchors and redo them. Settle these
up front, when only song.yaml exists and no GPU time has been spent:
| Decision | Where it lives | Why it has to be early |
|---|
| Aspect & resolution | video.resolution: [w, h] | Hard-coded into every anchor prompt ("vertical 9:16" vs "16:9 widescreen") and every render. This is LTX's working resolution; the LTX-2.3 spatial upscaler doubles it for the final output. Common picks: [832, 448] (16:9 broadcast → 1664×896 final), [448, 832] (9:16 vertical → 896×1664 final), [576, 1024] (9:16 a touch larger → 1152×2048 final). Both dims must be multiples of 32. Switching aspect after anchors invalidates every PNG. |
| Anchor scale | video.anchor_scale | Default 2.0 — flux2 anchors render at resolution × scale so they carry detail at the final upscaled video resolution. Larger anchors take longer to render but produce noticeably crisper output. Set 1.0 if you'd rather flux2 work at the working resolution (faster, looser final detail). |
| Cast | subjects: + reference photos in the project dir | Each named subject (operator, dancers, interviewer, …) needs (a) a one-line description token and (b) at least one reference image so per-scene i2i/i2i2 anchors can lock identity. Adding a character later means re-rolling every scene they appear in. |
| Canonical settings | sheets/<setting>.png (project convention) + scenes[].anchor.references | Pre-render one PNG per recurring location (e.g. main stage, b-stage, exterior). Scene anchors i2i2 against [character, setting] so the world stays visually constant. Without canonical setting refs, each scene anchor invents its own version of the place. |
| Lipsync mode | video.lipsync_audio | If you want LTX to lipsync against a vocal-forward remix instead of the mastered mix, supply it now. Switching mid-project means re-rendering every singing scene. |
| Number of suno variants | suno.runs (× 2 variants per run) | More variants = more parallel final_vN.mp4 outputs at assembly time, but every variant runs against the same scene visuals. Setting this up later is cheap; setting it down means you waste suno credits. |
| Quality gates | gate_confirm_song, gate_confirm_anchors | Default true (recommended). Flip to false only when you're fully confident; bad gate-skips waste hours of LTX time on bad anchors. |
Anything else (per-scene prompts, camera LoRAs, transitions on/off, individual
anchor tweaks) is cheap to iterate on later — those only re-render one scene
at a time and don't propagate.
Command index
Every step of the pipeline maps to one CLI subcommand in
scripts/music_video.py. The table below
cross-references each subcommand to its handler in code so authors can read
the exact behaviour, and lists every spec field the step consumes.
| Step | CLI | Handler | Reads from spec | Writes |
|---|
| 1 | init <slug> [--theme=<txt>] [--force] | cmd_init | (no spec; takes slug + --theme + --force) | <workspace>/<slug>/song.yaml skeleton |
| — | plan <spec> | cmd_plan | full spec (validation + breakdown) | stdout |
| 3 | song <spec> | cmd_song | title, style, lyrics, suno.runs, suno.make_instrumental | song.mp3, song_v2.mp3, …, song_meta.json |
| 7 | anchors <spec> | cmd_anchors → _generate_anchor per scene | anchor_image, anchor_prompt, subjects, scenes[].anchor.* | anchor.png (top-level if missing) + scenes/NNN-<label>-anchor.png × N |
| 9 | scene N <spec> | cmd_scene | full scene block, video.*, subjects | scenes/NNN-<label>.mp4, scenes/NNN-<label>-last.png, song_slices/NNN-<label>.mp3 |
| 9 | scenes <spec> | cmd_scenes (loops cmd_scene) | same | same × N |
| 9b | transitions <spec> | cmd_transitions | video.transitions.*, scenes[].transition_from_prev.* | scenes/NNN-<label>-transition.mp4 × (N−1) (only if video.transitions.enabled) |
| 10 | assemble <spec> | cmd_assemble | scenes[].duration_sec, video.transitions.*, tail_buffer_sec | final.mp4, final_v2.mp4, … (one per song variant) |
| * | all <spec> [--no-gate] | cmd_all | gate_confirm_song, gate_confirm_anchors (+ everything above) | runs song → anchors → scenes → transitions → assemble, stops at the two gates by default |
| — | status <spec> | cmd_status | scenes list (to enumerate expected outputs) | stdout report (✓/—) per file |
Quality gates — cmd_all stops after song (gate 1) and after anchors (gate 2) by default. Override per-spec via gate_confirm_song: false / gate_confirm_anchors: false, or per-invocation via --no-gate.
Workflow-step column (1, 3, 7, 9, 10) — corresponds to the numbered phases in The loop above. Steps 2/4/5/6/8 are human review/decisions (variant pick, STT, scene authoring, anchor review), not CLI commands.
Using with openclaw agents
Any agent with music-video in its agents.list[<id>].skills array can run
this skill end-to-end. The gates make it safe to delegate: the agent runs
all, waits at gate 1, and posts the variants back for a human review.
After renaming, the agent is prompted again and runs all a second time to
pass gate 2, and so on.
Example — asking Athena (char6166r) to start a project:
openclaw agents run char6166r --message "Start a music video project: create a song about <theme>, init it in your workspace, and launch the suno generation."
Athena will init, fill the YAML, run song, then report the variants back
(gate 1). After a human renames the chosen variant, Athena resumes with
scene authoring → anchors → gate 2 → scenes → assemble.
Long renders MUST go through sessions_spawn — never block the agent
scenes, transitions, all for any non-trivial spec take 30 minutes
to several hours. The agent invoking those commands must NOT exec them
directly and tail stdout — that locks the agent's turn for hours, floods
the chat with [18:21:43] transition 2→3 done lines, and prevents any
further user message from being handled.
The right pattern is OpenClaw's sub-agent mechanism:
sessions_spawn (or /subagents spawn) creates a background child run
with push-based completion. The parent returns to the user within
seconds; the child runs the long subprocess; when the child finishes,
OpenClaw wakes the parent with the result. Do not poll sessions_list
or /subagents list — the announce is automatic.
Skeleton the agent should follow (pseudocode for the parent's tool call):
sessions_spawn({
agentId: "<self>", // same agent, isolated child session
context: "isolated", // child gets the brief only — no transcript fork
task: """
Run the music-video scenes phase for /workspace/<slug>:
uv run --script /home/sandbox/.openclaw/skills/music-video/scripts/music_video.py \\
scenes /workspace/<slug>/song.yaml
Stream nothing back during the run. When complete, report:
- exit code
- the count of files in <slug>/scenes/*.mp4
- the path to <slug>/run.log
""",
runTimeoutSeconds: 21600, // 6 h ceiling — adjust per spec size
})
The parent immediately tells the user "Render started in the background,
I'll let you know when it's done" and yields its turn. The next visible
parent turn is the announce when the child finishes (success or fail).
Same pattern for transitions, all, and assemble when the spec is
large. The short-running commands (init, plan, song, anchors,
single scene N) stay inline — they finish within an agent turn budget.
For raw subprocess detachment without spawning a child agent (you don't
need it here, but it's the same shape), see OpenClaw's background
tasks — nohup, setsid, or disown on a wrapped
command, with the agent later inspecting the run.log to surface status.
Script paths (absolute; no ~)
The canonical install path is ~/.openclaw/skills/music-video/scripts/music_video.py.
~/.openclaw/… works in an interactive shell but NOT always under exec, so
when invoking from another agent or process pass an absolute path:
- OpenClaw sandbox:
/home/sandbox/.openclaw/skills/music-video/scripts/music_video.py
- Host install:
$HOME/.openclaw/skills/music-video/scripts/music_video.py
- Fresh clone:
<repo>/music-video/scripts/music_video.py
The script uses a uv run --script shebang with inline deps (pyyaml,
imageio-ffmpeg). Invoke it via uv run --script so deps auto-install:
uv run --script /path/to/music-video/scripts/music_video.py <cmd> ...
Plain python3 music_video.py … will fail with ModuleNotFoundError: yaml
unless you've already installed pyyaml in your env. Use uv instead — it's
the designed path.
Commands
music_video.py init <slug> [--theme=<text>] [--force]
music_video.py plan <spec.yaml>
music_video.py song <spec.yaml>
music_video.py anchors <spec.yaml>
music_video.py scene N <spec.yaml>
music_video.py scenes <spec.yaml>
music_video.py transitions <spec.yaml>
music_video.py assemble <spec.yaml>
music_video.py all <spec.yaml> [--no-gate]
music_video.py status <spec.yaml>
Typical for a 3-scene 60-second video: ~6 minutes wall time (song ~4 min, scenes ~45s each sequential, assembly a few sec).
YAML spec
Minimum viable:
title: Glass Harbour
style: "folk noir, coastal Americana, 74 BPM, fingerpicked acoustic guitar, bowed upright bass, sparse brushed snare, foghorn field recording, salt wind ambience, weathered male vocal, close-mic room reverb, mournful tone"
lyrics: |
[Verse]
Salt in the air and rust on the crane,
the fishermen leave before the morning train.
[Chorus]
Glass harbour, glass harbour,
where the cold light bends.
video:
fps: 24
resolution: [1024, 576]
negative: "pc game, cartoon, modern tech, text, ugly"
crossfade: 0.5
camera_lora_strength: 0.8
fast: true
anchor_image: anchor.png
scenes:
- label: establishing
start_sec: 0
duration_sec: 8
prompt: "wide fog-shrouded harbour at dawn, lonely crane silhouette, salt spray, muted palette"
image: "@anchor"
- label: fisherman
start_sec: 8
duration_sec: 10
prompt: "weathered fisherman hauling nets, breath visible, early light"
image: "@last"
- label: lighthouse
start_sec: 18
duration_sec: 8
camera_lora: static
prompt: "lighthouse blinks through fog, beam sweeps across waves"
image: "@last"
Complete schema reference
Every key the orchestrator reads from a spec, cross-referenced to the
function in scripts/music_video.py that consumes
it. Anything not listed is silently ignored.
Top-level
| key | type | default | code | meaning |
|---|
title | str | "Untitled" | cmd_song | suno generation title |
style | str | "" | cmd_song | suno producer-style brief |
lyrics | str | "" | cmd_song | suno lyrics with [Verse]/[Chorus] tags |
suno.runs | int | 1 | cmd_song | each run yields 2 variants → runs * 2 mp3s |
suno.make_instrumental | bool | false | cmd_song | request instrumental-only variants |
subjects | dict[str, str] | {} | _expand_subjects | {name} tokens substituted in every prompt (scene + anchor) |
anchor_image | path | none | cmd_anchors | top-level PNG; scenes referencing image: "@anchor" use this |
reference_sheet | path | none | cmd_scene | project ingredients reference sheet (black-bg element panels); used by scenes with mode: ingredients. Author with storyboard/generate_reference_sheet.py. Make its aspect match video.resolution. |
anchor_prompt | str | falls back to title + style | cmd_anchors | flux2 prompt for the top-level anchor when anchor_image is missing |
gate_confirm_song | bool | true | cmd_all | all halts after song for human review |
gate_confirm_anchors | bool | true | cmd_all | all halts after anchors for human review |
video | dict | {} | _video_spec | renderer config (see below) |
scenes | list | [] | cmd_plan | scene list (rendered in order) |
video.*
| key | type | default | code | meaning |
|---|
video.fps | int | 24 | _video_spec | LTX/output frame rate |
video.resolution | [w, h] | [1024, 576] | _video_spec | LTX working resolution (final video is 2× this — see anchor_scale); both dims must be multiples of 32 |
video.anchor_scale | float | 2.0 | _generate_anchor | flux2 anchor renders at resolution × anchor_scale. Default 2.0 matches LTX-2.3's spatial upscaler so anchors carry real detail at the final output resolution. Set to 1.0 for legacy behaviour (anchor at working res). |
video.negative | str | None | _video_spec | negative prompt applied to every LTX scene |
video.tail_buffer_sec | float | 0.0 | _video_spec | extra seconds rendered past EVERY scene's duration_sec, trimmed at assembly. Two reasons: (a) lipsync phoneme look-ahead, (b) LTX's ×8 latent temporal compression rounds frame counts DOWN — without this buffer a 4.21s request can land at 4.04s and audio drifts ahead of video. Recommend 0.5 for all music videos. |
video.lipsync_audio | path | none | cmd_scene | vocal-forward remix used for ia2v conditioning only; song.mp3 stays canonical for assembly |
video.camera_lora | str | none | cmd_scene | default camera LoRA for scenes that don't set their own |
video.camera_lora_strength | float | 0.8 | cmd_scene | default LoRA strength |
video.fast | bool | false | cmd_scene | default for scene[].fast (skip 2-pass refine; iteration shortcut) |
video.hdr_lora | dict | none | _normalize_loras | LEGACY single-LoRA slot, auto-converted to a 1-entry loras: list at render. Keys: file, strength, `reference_video |
video.loras | list | none | _normalize_loras | NEW chained LoRA schema. Each entry: `{file, kind: ic_lora, strength, reference_video |
video.base_guide_strength | float | 0.9 | cmd_scene | LTX base-pass guide strength for multiguide scenes |
video.refine_guide_strength | float | 0.7 | cmd_scene | LTX refine-pass guide strength for multiguide scenes |
video.transitions | dict | none | cmd_transitions | per-boundary LTX morph clips (see below) |
video.transitions.*
| key | type | default | code | meaning |
|---|
enabled | bool | false | cmd_transitions | turn the transitions stage on |
duration | float | 2.0 | cmd_transitions | total per-boundary clip length (seconds); 0 = hard cut |
guide_sec | float | 1.0 | cmd_transitions | real-video guide on each side; middle = duration − 2·guide_sec |
fps | int | inherits video.fps | cmd_transitions | transition fps |
prompt | str | "smooth morph" | cmd_transitions | default morph prompt; overridable per boundary |
default_b_sparse | str | "96" | cmd_transitions | comma list of latent positions for B-side anchor (e.g. "72,80,88,96" for 4f into singing) |
fast | bool | inherits video.fast | cmd_transitions | skip 2-pass refine on transition renders |
scenes[] items
| key | type | default | code | meaning |
|---|
label | str | "scene" | _scene_stem | short id; used in output filenames (NNN-<label>.mp4) |
start_sec | float | required | cmd_scene | scene start in song; drives audio-slice start |
duration_sec | float | required, ≤15s | cmd_scene | scene length; hard cap is MAX_SCENE_DURATION (15.0s) |
prompt | str | required | cmd_scene | LTX ia2v prompt (continuous shot; {subjects} tokens expand) |
image | str | "@anchor" | _resolve_image | first-frame source: "@anchor", "@last", "@none" (forces t2v), or a literal path |
anchor | dict | none | _generate_anchor | per-scene flux2 pre-render config (see below); when present, this PNG becomes the scene's first frame |
mode | str | none | cmd_scene | set to ingredients to render this scene via the LTX-2.3 reference-sheet IC-LoRA instead of ia2v/t2v (see "Ingredients mode" below). Inherits video.mode. |
reference_sheet | path | inherits top-level | cmd_scene | per-scene sheet override for mode: ingredients |
ingredients_lora_strength | float | 1.4 (ltx2 default) | cmd_scene | ingredients LoRA strength for this scene |
ingredients_reference_strength | float | 1.0 (ltx2 default) | cmd_scene | LTXAddVideoICLoRAGuide strength for this scene |
guides | list | none | cmd_scene (resolved via storyboard.lib.guides.resolve_guides) | mid-scene LTXVAddGuide entries (see "Multi-guide scenes") |
camera_lora | str | from video.camera_lora | cmd_scene | one of static, dolly-in, dolly-out, dolly-left, dolly-right, jib-up, jib-down |
camera_lora_strength | float | from video.camera_lora_strength | cmd_scene | LoRA strength override |
fast | bool | inherits video.fast | cmd_scene | skip 2-pass refine for this scene |
hdr_lora | dict/null | inherits video.hdr_lora | _normalize_loras | per-scene LEGACY LoRA override; null disables. Prefer per-scene loras: below. |
loras | list/null | inherits video.loras | _normalize_loras | per-scene chained LoRA override. Same shape as video.loras. |
lipsync_audio | path/null | inherits video.lipsync_audio | cmd_scene | per-scene ia2v audio override; null forces song.mp3 |
base_guide_strength | float | inherits video.base_guide_strength | cmd_scene | multiguide base-pass guide strength override |
refine_guide_strength | float | inherits video.refine_guide_strength | cmd_scene | multiguide refine-pass guide strength override |
transition_from_prev | dict | none | cmd_transitions | override the LTX morph clip on the INCOMING boundary (see below) |
scenes[].anchor.*
When anchor.prompt is set, _generate_anchor flux2-pre-renders a PNG per scene and uses it as that scene's first frame, overriding image.
| key | type | default | code | meaning |
|---|
type | t2i | i2i | i2i2 | i2iN | angles | inferred from len(references) | _generate_anchor | flux2 mode |
prompt | str | required | _generate_anchor | flux2 prompt; {subjects} tokens expand |
reference | path | none | _generate_anchor | single reference image (for i2i) |
references | list[path] | none | _generate_anchor | 2+ references (for i2i2, i2iN, angles); first is the primary |
width | int | from video.resolution[0] | _generate_anchor | anchor render width override |
height | int | from video.resolution[1] | _generate_anchor | anchor render height override |
steps | int | flux2 default (~8) | _generate_anchor | flux2 sampler steps |
keep_identity | bool | true | _generate_anchor | for i2i/i2i2: auto-append "keep face/features from reference" guard. Set false to suppress |
angle_prompts | list[str] | [prompt] | _generate_anchor | for type: angles — multi-pose batch from one reference |
Anchor-type cheatsheet (full discussion in the storyboard skill):
| type | refs | use when |
|---|
t2i | 0 | pure environment / no character (setting shots) |
i2i | 1 | character into a specific scene (most common) |
i2i2 | 2 | character + setting blend, or character A + character B meet |
i2iN | 3+ | small group scenes (cap at 3–4 — identity drifts past that) |
angles | 1 | character-sheet building (multi-pose batch from one reference) |
scenes[].guides[] items (multi-guide)
Resolved via storyboard.lib.guides.resolve_guides.
| key | type | default | meaning |
|---|
image | str | required | @anchor, @last, or a path; relative paths resolve against project dir |
at_sec | float | — | absolute seconds from scene start (takes precedence) |
at_relative | float (0..1) | — | fraction of duration_sec |
at_frame | int | — | explicit LTX latent frame (snaps to multiples of 8) |
strength | float | 1.0 | LTXVAddGuide weight |
label | str | — | human-readable, ignored by resolver |
scenes[].transition_from_prev.*
Goes on the INCOMING scene (B-side), not scene A. Read by cmd_transitions.
| key | type | default | meaning |
|---|
duration | float | from video.transitions.duration | per-boundary length; 0 = hard cut (LTX morph skipped) |
b_sparse | str | from video.transitions.default_b_sparse | B-side latent positions for this boundary |
prompt | str | from video.transitions.prompt | morph prompt for this boundary only |
ic_lora | dict | none | IC-LoRA structural reference for the morph's empty middle. Single-ref only today (see "Multiguide stitching" below) |
The ic_lora dict shape:
transition_from_prev:
duration: 4.0
ic_lora:
file: ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors
strength: 1.0
reference_video: cond/22-23-spin.mp4
reference_strength: 1.0
frame_idx: 24
reference_video is auto-trimmed by cmd_transitions if it would
overflow (length_frames - frame_idx) / fps — the assertion
latent_idx + guide_latent.shape[2] <= latent_length in
LTXAddVideoICLoRAGuide.execute is enforced upstream. A trimmed copy is
written next to the transition mp4 as …-ictrim.mp4.
Ingredients mode (project-level character/prop/location consistency)
When a music video must keep the same character, props and locations
consistent across every scene, condition each scene on ONE project reference
sheet (black-bg element panels) via the LTX-2.3 ingredients IC-LoRA, instead of
authoring a per-scene flux2 anchor.
-
Build the sheet once with the storyboard skill:
storyboard/scripts/generate_reference_sheet.py --out sheet.png \
--character cast.png --character-desc "…" \
--prop "x=…" --location "…" \
--width 1024 --height 576
-
Point the project at it and flag the scenes:
reference_sheet: sheet.png
video: { resolution: [1024, 576], fps: 24 }
scenes:
- label: chorus-1
start_sec: 41.0
duration_sec: 5
prompt: "the singer on the neon rooftop, holding the mic, rain"
mode: ingredients
mode: ingredients is checked first in the scene dispatch (authoritative
over image/guides). The render is silent by design — the song is muxed
over the whole timeline at assemble time and assemble.py strips per-clip audio
anyway, so no audio slice is conditioned. The scene's generic video.negative is
dropped in favour of the ingredients LoRA's tuned negative (set a per-scene
negative: to override). Tune with ingredients_lora_strength (def 1.4) /
ingredients_reference_strength (def 1.0). Trained buckets are landscape
(768×448, 960×544); portrait is off-bucket — evaluate. Compose freely with
camera LoRAs and transitions. See the comfyui skill's ingredients command and
storyboard's "Reference sheets" section.
Camera LoRAs
Seven cinematic motion LoRAs are installed on the comfy server and can be
applied per-scene via camera_lora: <name>:
| name | effect |
|---|
static | locks camera still (good for contemplative shots) |
dolly-in / dolly-out | push in / pull back |
dolly-left / dolly-right | lateral tracking |
jib-up / jib-down | vertical crane |
Defaults to no LoRA if unset. Strength via camera_lora_strength (top-level
default 0.8; overridable per-scene). Pair the LoRA name with a matching prose
description of the same move in the prompt — the LoRA reinforces the intent.
Scene transitions (LTX, not ffmpeg)
Boundaries between @last-chained scenes usually land with a visible seam —
ia2v conditioning drifts enough that adjacent scenes don't quite meet. The
transitions stage renders a short LTX clip per boundary that morphs from
scene A's tail into scene B's head, using real video guides from both
sides (not still frames) so the motion continues across the cut.
Enable + tune in video.transitions:
video:
transitions:
enabled: true
duration: 4.0
guide_sec: 1.0
fps: 24
default_b_sparse: "96"
The resulting shape for a 4.0s / 1.0s boundary is 1s A-guide → 2s masked
morph (audio drives the invention) → 1s that ends on scene B's head.
Assemble overlaps the transition with scene A's trailing duration/2s and
scene B's leading duration/2s so the final timeline stays bit-exact with
the song.
Per-boundary override goes on the INCOMING scene (not scene A) via
transition_from_prev:
scenes:
- label: chorus_in
transition_from_prev:
duration: 4.0
b_sparse: "72,80,88,96"
prompt: "…optional override for this boundary's morph…"
Hard cut: set duration: 0 to skip the LTX transition for one
boundary. The assemble step butt-cuts straight from prev scene's tail
to this scene's head — useful on drum-hit / chorus-crash entries
where you want the impact, not a smooth morph. Pair with a punchy
visual change (palette flip, location jump) for maximum bite.
scenes:
- label: chorus_crash
transition_from_prev:
duration: 0
Defaults that work: 4s duration, 1s guide, default_b_sparse: "96" (1f).
4f ("72,80,88,96") was empirically the sweet spot for boundaries into
singing shots — more anchors prevent identity drift when the character
immediately has to sing. Contiguous multi-frame B-blocks (e.g. 16 frames at
positions 80-96) caused a mid-transition freeze at snap-in; always keep
the B-side sparse (every 8 latent frames).
b_sparse positions that map to BEFORE scene B starts are dropped, not
clamped. A 4s transition between c2_breath (ends 130.02s) and tunnel_drop
(starts 132.0s) puts the boundary centre at 130.02 → transition spans
[128.02, 132.02]. Latent positions 72/80/88 then map to song times
131.02 / 131.35 / 131.69 — all BEFORE scene B's start at 132.0 → negative
src_frame in next_video. The old max(0, x) clamp silently locked
all four anchors to scene B's frame 0, killing the morph's freedom in the
empty middle (every output snapped to scene B's static head by latent 72).
The current behaviour skips those positions with a stderr warning and
proceeds with whatever anchors remain. For boundaries where scene B
starts in the back half of the transition window, override b_sparse: "96"
explicitly to keep just the tail anchor.
Multiguide stitching (the canonical seamless-transition pattern)
When a morph should bridge two scenes with continuous structural motion
through the empty middle (not just smooth-pixel-interpolate from A's tail to
B's head), use the multiguide stitch: A-side LTXVAddGuide at
frame_idx=0 + IC-LoRA cond at frame_idx=24 + single B-side anchor at
latent 96. The IC-LoRA's structural reference video carries continuous
geometry through the morph's empty middle, the morph LoRA blends the
endpoints, and the single tail anchor lands scene B cleanly without
yanking the latent.
Recipe:
-
Render a time-aligned cond for the transition's song-time window
via midi2canny.py --polyfield-only (or tunnel_3d.py, etc.). For a
boundary spanning song time [T_a, T_b]:
midi2canny.py --polyfield-only \
--drums midi/drums_audio_aligned.mid \
--vocals midi/vocals_audio_aligned.mid \
--guitar midi/guitar_audio_aligned.mid \
--start-sec T_a --duration (T_b - T_a) \
--fps 24 --width <w> --height <h> \
--poly-count 24 --poly-rings 3 \
--output cond/N-N+1-trans.mp4
Audio-aligned MIDIs are critical — the polyfield's drum-hit scale
pulses and vocal-pitch rotV impulses then line up with what the LoRA
will be conditioning the audio against. A cond rendered for the wrong
song-time window produces an output that "looks the same every render"
because the cond's temporal arc doesn't reinforce the audio's arc.
-
Wire the cond into the per-boundary transition_from_prev.ic_lora
block on scene B with frame_idx: 24 (= 1.0s @ 24fps = the empty
middle's start). The non-zero frame_idx is critical: a guide at
frame_idx=0 collides with the A-side LTXVAddGuide's keyframe
time-start, which makes comfy_extras.nodes_lt.LTXVCropGuides
undercount num_keyframes (it uses torch.unique(time_starts)) and
pass-2 refines the un-stripped guide frames as content — 5-9s glitched
output instead of clean 4s. frame_idx: 24 puts the cond past the
A-side block.
-
Override b_sparse: "96" if scene B starts in the back half of
the transition window (see warning above). Single tail anchor only —
gives the morph LoRA the full empty middle to follow the cond.
-
Match scene B's IC-LoRA cond to its own song-time window. Scene 23
spans 132-139s, so its hdr_lora.reference_video should be a cond
rendered with --start-sec 132 --duration 7, NOT the transition's
cond (which covers 128-132s). The output looks the same every render
if the cond's temporal arc doesn't match the audio's arc.
-
If scene B's intro is static, scene B has to be rerendered with an
action prompt. The morph LoRA interpolates A → B; if B's first
frames are a held shot, the morph lands on a held shot. Rewriting
scene B's prompt to "action from frame one" + bumping its IC-LoRA
reference_strength is what unlocks "polygons whipping past the face"
instead of "face holding still until the drum hit at +2.8s".
Worked example for the stable-altitude v9 c2_breath → tunnel_drop boundary:
- label: tunnel_drop
start_sec: 132.0
duration_sec: 7.0
transition_from_prev:
duration: 4.0
b_sparse: "96"
ic_lora:
file: ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors
strength: 1.0
reference_video: cond/22-23-spin.mp4
reference_strength: 1.0
frame_idx: 24
hdr_lora:
file: ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors
strength: 1.0
reference_video: cond/23-tunnel_drop-v2.mp4
reference_strength: 1.0
prompt: >
{operator}'s face flickering at the center of an EXPLODING geometric
mandala of neon-cyan + sodium-orange polygons that spin and shatter
outward from frame one, polygons whipping past the face, … continuous
violent motion through the drum slam at 2.8s into the explosion peak.
Multi-reference IC-LoRA (multiple LTXAddVideoICLoRAGuide nodes chained
under different cond videos) is NOT supported today. Upstream
comfy_extras.nodes_lt.LTXVCropGuides.get_keyframe_idxs counts
num_keyframes = unique(keyframe_idxs[:, 0, :, 0]) — when 2+ guides share
any pixel-frame time-start, the unique-count undercounts and pass-2
refines the extras as content. Workaround: only one IC-LoRA per
transition. The transition_from_prev.loras (list) schema is reserved
but rejected at yaml-load if used.
Multi-guide scenes (guides: field)
A scene whose first frame can't/shouldn't show the character — e.g. opens
on an empty landscape or a prop before the character enters — needs a
mid-scene character anchor or LTX will invent a random identity. Add a
guides: list with pre-rendered flux anchors at specific times:
scenes:
- label: meets_hare
start_sec: 98.110
duration_sec: 13.630
image: "sheets/mid_anchors/hare_candle_first_00001_.png"
prompt: "Wide dusk-meadow — a candle on a stone; {hare} cartwheels around it; {snakebird} shields the flame…"
guides:
- image: "sheets/mid_anchors/hare_character_00001_.png"
at_relative: 0.5
strength: 1.0
Each guide entry: image (literal path, @anchor, or @last) + one of
at_sec / at_relative (0..1) / at_frame + optional strength (default 1.0).
Frame indices snap to multiples of 8 (LTX latent quantization). For long
shots (13s+), add two guides (e.g. at 0.5 and 0.9) to hold identity through
the whole shot — a single mid-anchor drifts by the tail.
When guides: is present, the scene renders via LTX multiguide (chained
LTXVAddGuide). The shared implementation lives in the storyboard skill
(lib/guides.py) so drama-video and one-off renders use the same resolver.
A fuller reference with all optional fields is at references/example.yaml. Format details in references/song-format.md.
Folder layout (auto-created)
<project>/
song.yaml
song.mp3
song_meta.json
song_slices/
001-establishing.mp3
002-fisherman.mp3
003-lighthouse.mp3
scenes/
001-establishing.mp4
001-establishing-last.png
002-fisherman.mp4
002-fisherman-last.png
003-lighthouse.mp4
003-lighthouse-last.png
final.mp4
run.log
<project> = the directory containing song.yaml. The script writes everything there.
Prompting recipe (for agents filling the YAML)
- Style brief is a producer-style text-to-music prompt. See
~/.openclaw/skills/suno-mcp/references/style-guide.md for the full pattern (genre, BPM, instruments, production, vocal, mood, narrative sentence). Avoid keyword lists.
- Lyrics use
[Verse]/[Chorus]/[Bridge]/[Instrumental] tags. One line per phrase; no mid-line punctuation. See ~/.openclaw/skills/suno-mcp/references/lyrics-guide.md.
- Scenes should map to lyric structure. For a ~3-minute song, prefer shorter, more numerous scenes (~6-8s each, 20-30 total) over a small number of long ones. Long scenes (15s+) are at LTX's edge and benefit less from re-rolls. Short scenes give you variety per chorus and let you place dancer close-ups / cutaway beats between operator-lipsync passes. Align
start_sec/duration_sec with the actual song structure — duration accuracy matters for audio-reactive feel.
- Scene prompts should describe what's on screen — not tell a story. LTX-2.3 handles short camera moves well (dolly, pan, push-in) and struggles with cuts. Keep each scene as a continuous shot.
- Image chain with
@last for continuity between adjacent scenes; use @anchor or a literal path to reset to a different look (verse/chorus transitions).
- Resolution: 1024×576 is a good default for music videos. 768×512 is faster. Higher eats GPU. (LTX's 2× upscaler then doubles to the final output — see
video.anchor_scale.)
- Negative prompt: use the
video.negative field to blacklist "ugly, pc game, cartoon, text" etc. Keeps LTX-2.3 from drifting.
- Cold-open / first appearance. Don't put the protagonist in scene 1 by default. A short atmospheric
t2i / i2i (curtain, empty stage, exterior establishing) for the instrumental intro, then cut to the protagonist's first appearance exactly when a defining instrument enters (typically the drums — use analyze_midi.py to find the first drum hit). The cut feels earned because the protagonist's entrance lands with the beat that just kicked in. The cold-open boundary is almost never the first vocal — it's the first drum hit, which is usually 1-3s earlier.
- Variety beats inside each chorus. A chorus rendered as ONE 15s scene of the protagonist lipsyncing reads as static. Split it into 3 shorter (4-7s) scenes: wide-stage establishing, close-up of a different character (a dancer mouthing the answer hook), push-in on the protagonist's face. Same lyric, three different camera languages. This is the single biggest "professional music video" lever short of using transitions.
- Featured-character cameos (call-and-response, duets, eye-contact moments). When the lyric is about engagement — call-and-response, "I wanna hear you say", a partner's answer — having the operator literally turn and meet eyes with another person makes the song's emotional argument visible. Two patterns:
i2i2 against [operator, group_setting] picks the partner from the recurring cast (a chorus dancer). LTX renders an interchangeable face — fine for chorus formations, bad for "the singular partner" moment.i2iN against [operator, setting, featured_singer.png] uses a dedicated reference image for the duet partner so they're visually distinct from the recurring cast. Make featured_singer.png a t2i upfront with explicit "different from any of the dancers" language (different hair, different outfit, different posture). Reuse this ref across every eye-contact / duet scene to keep her identity consistent.
- Hand anatomy guidance for crowd shots. flux2 will hallucinate extra hands the moment multiple people are mid-gesture in the same frame ("operator with mic raised + dancers with arms up" → often produces a third hand on the operator). Write the anchor prompt with explicit positional language per character (e.g. "operator's right hand thrusting the chrome mic forward, his left hand at his side; dancers behind with both arms straight up overhead in symmetric V") and add
anatomically correct hands to the prompt tail. Same fix for "operator + featured singer sharing a mic" — say which hand holds the mic.
- Avoid long sustains at scene cuts. Scenes that end mid-sustain (held "ever", held "do", held "say") cut LTX's lipsync mid-phoneme. Move the boundary to after the sustain ends — check
_words.srt for the exact word-end time, not the segment-end time. This is the single most common source of "the lipsync looks weird at the cut" reports.
Troubleshooting
"song.mp3 missing" when running scene → run song first, or use all.
"scene N: @last requested but ...-last.png does not exist" → previous scene hasn't been generated yet; run them in order or use all.
Final video's audio feels off → each scene is conditioned on its audio slice, but the final pass overlays the clean suno track. If a scene's video is mis-synced, regenerate that scene (scene N spec.yaml) — suno audio is not re-generated, just the video.
Suno login required → mcporter call suno.suno_login --config ~/.openclaw/config/mcporter.json first.
Scenes visually inconsistent → pin @anchor on the first scene and chain @last through the rest. Add specific character/place descriptors in every scene prompt, not just the first.
UnicodeEncodeError on Windows → the script prints unicode glyphs (→ ✓ ⚠) and writes them to run.log. As of the latest version, music_video.py reconfigures stdout/stderr/log file to utf-8 at startup, so this should "just work" on PowerShell / cmd.exe. If you're on an older copy and still see 'charmap' codec can't encode character '→', prefix the invocation with PYTHONIOENCODING=utf-8 or upgrade the skill.
Lipsync feels off / words cut at scene boundaries → your start_sec/duration_sec are landing inside word spans rather than at word ends. Don't trust segment SRT timings for this — vocalists sustain final phonemes past line-end. Open <prefix>_words.srt, find what word is being sung at each scene's end_sec, and shift the boundary to after that word's end. Re-render only the affected scene + the one starting at the new boundary; everything else is preserved by the restart-safe scene cache. See "Avoid long sustains at scene cuts" in the prompting recipe.
Cuts feel "off the beat" → your boundaries are not bar-aligned. Run analyze_midi.py <project> to get the bar grid (e.g. 123.4 BPM → 1.944s/bar), then nudge each end_sec to the nearest bar boundary that also respects a word-end (the two usually coincide within ~150ms for well-recorded vocals).
mido.KeySignatureError: Could not decode key with 16 sharps → Suno's stem-pack midi exports occasionally emit key signature meta events outside the standard MIDI spec. The current analyze_midi.py / analyze_song.py pre-populate mido's decode table with safe fallbacks so the file loads — if you're on an older copy, monkey-patch mido.midifiles.meta._key_signature_decode to add fallback entries before the first MidiFile() call.
Advanced (not in v1)
- Beat detection & waveform-linked light effects → not implemented. If needed, add a separate post-process pass with ffmpeg's
showwaves / asetnsamples + blend modes, fed by a beat track from librosa.onset.onset_strength.
- Lyric subtitles → not generated. Add an SRT track separately.
