create-vo-anchored-beat-video

name

create-vo-anchored-beat-video

description

Produce a VO-driven 9:16 / 1:1 / 16:9 shortform explainer video where every visual moment is snapped to a VO word boundary. Renders the voiceover first, extracts Whisper word-timestamps, decomposes the script into N "beats" (one HTML hyperframe per visual station + B-roll for moving footage), renders each beat deterministically via `window.renderAt(t)`, then concats + mixes audio + burns captions on the master. Works for listicle, mechanism explainer, stat-bombing, myth-buster, cost-math, narrative monologue, document-reveal, or any script-first shortform. Does NOT produce live UGC selfie-cam, music-video, or on-camera doctor talking-head (those need different molecules). Orchestrator-style with 4 human gates and a resumable phase pipeline. Validated on Everself v4 (33 beats, 65.88s, McGowan ESG listicle).

create-vo-anchored-beat-video

Purpose

Produce a shortform explainer video whose visuals are word-precisely anchored to a pre-rendered voiceover. The molecule's core claim:

Every beat boundary snaps to a Whisper word boundary on the rendered VO. When the VO says "no scars," the NO SCARS slate is on screen for exactly that 0.80s.

This is achieved by rendering the VO first (not last), extracting word-level timestamps, then authoring one HTML hyperframe per script "station" (a sentence, a stat, a callout, a transition) with a deterministic window.renderAt(t) renderer driven frame-by-frame by Playwright. Beat durations are derived from Whisper timestamps, not authored arbitrarily.

Use when the user wants any of:

• Listicle ("5 things about X")
• Mechanism explainer ("how X actually works in 60 seconds")
• Stat-bombing / data ad
• Myth-buster / Q&A ("MYTH / FACT" pairs)
• Cost-math comparison ad
• Narrative monologue (founder story, patient testimony with VO)
• Document-reveal ("The Receipt" credentialing ad)
• Any 30-90s script-first shortform with mixed motion-graphics + B-roll

Do not use for:

• Live UGC selfie-cam where the speaker is on camera (no script-to-render-first) → use molecules/ugc-ad/
• On-camera doctor talking-head with lipsync → use atoms/lipsync/create-lipsync-hedra (a different molecule should wrap that)
• Music-video format where a song drives, not VO → use music-video-ad/ molecules
• Photoreal product spots → use molecules/product-video/create-product-sizzle-reel or create-video-veo3

Inputs

Required (in --script invocation arg):

• script.md — locked VO script (markdown). May include [delivery] style tags at the start. Word count budget: ~3 words per second of target duration (eleven_v3 + atempo=1.20-1.25 = ~180wpm effective). See feedback_elevenlabs_v3_wpm_reality.md memory.

Collected by the orchestrator in Phase 0 (intake questionnaire):

1. Voice ID — from VOICES.md approved list, or a custom ElevenLabs voice. Required.
2. Delivery style — Doctor-Educator-Explainer / Friendly-UGC / Authoritative-News / Conversational / Other. Maps to ElevenLabs voice_settings (stability, similarity, style) + atempo.
3. Visual style — goose-graphics style slug (isometric-sage, matt-gray, etc.) OR path to a reference image to extract from. Drives palette, typography, hyperframe scaffolding.
4. Aspect ratio — 9:16 (default), 1:1, or 16:9.
5. B-roll budget — total B-roll count OR ratio (e.g. "8 B-rolls" / "70:30 MG:BR"). Pacing memory: fast-cut Reels want 25-30% B-roll for variety.
6. B-roll sources — Pexels search keywords per beat archetype + paths to any existing source videos to reuse (brand footage, prior shoots).
7. Music — ElevenLabs music prompt (genre + BPM + mood) OR "skip" for VO-only OR path to an existing track. Memory: ambient instrumental with cinematic atmosphere for explainer ads; music with vocals preferred for product/lifestyle.
8. End card — CTA copy + URL/comment-prompt + brand mark.
9. Output project folder — where to drop canonical 5-folder layout (see PROJECT_STRUCTURE.md).

Optional:

• Existing Whisper word-timestamps JSON (if VO already rendered) — orchestrator skips Phase 1+2 if provided.
• A goose-graphics DESIGN.md path if the style isn't in the central catalog yet.
• A prior beat manifest (resume from Phase 3).

Composed Atoms

Voiceover (Phase 1):

• atoms/voiceover/format-script-to-style — apply delivery style tags + word-count budget check.
• atoms/voiceover/generate-voiceover-elevenlabs-v3 — render with eleven_v3 + voice_settings.
• atoms/voiceover/create-voiceover-elevenlabs — fallback to standard model if v3 unavailable.

Transcription (Phase 2):

• atoms/source/transcribe-audio-fal — Whisper word-timestamps via FAL.
• atoms/source/split-audio-by-words — fallback chunking.

Beat decomposition (Phase 3):

• atoms/planning/create-motion-graphics-storyboard — supports the beat-level storyboard authoring step.
• atoms/planning/create-storyboard-html — renders storyboard.html so the user can review the proposed beats before render.

B-roll sourcing (Phase 4):

• atoms/clip-discovery/find-clips-pexels — search + download Pexels candidates per beat.
• atoms/editing/trim-video-clips — trim each B-roll to exact beat duration + ss offset.
• atoms/color/grade-consistency-pass — apply brand palette grading to all B-rolls.

Hyperframe authoring (Phase 5):

• atoms/motion-graphics/create-goose-graphics — pull the chosen style's DESIGN.md.
• atoms/motion-graphics/create-motion-graphics-hyperframes — author each beat HTML using the _shared.css / _shared.js + initRenderer(duration, renderFn) scaffold; render each beat to mp4 via Playwright frame-stepping at 25fps.
• atoms/overlays/render-video-overlay-track — composite static or fade-in overlays on top of B-roll beats (e.g. the "~70% CAPACITY" pill in the Everself v4 case study).

Music + audio (Phase 7):

• atoms/music/create-music-elevenlabs — generate ambient bed at target duration.
• atoms/audio-editing/mix-master — mix VO + music with normalize=0 (per feedback_ffmpeg_amix_normalize.md).

Assembly + captions (Phase 8):

• atoms/assembly/stitch-videos-ffmpeg — concat all beat mp4s into the silent master at uniform 25fps. CRITICAL: all beats must share framerate or concat silently drops frames (validated failure mode in Everself v4 build).
• atoms/captions/retime-captions-to-words — build an ASS file from Whisper words, scoped to B-roll windows only.
• atoms/captions/add-captions-burn — burn the ASS file on the final composite.
• atoms/captions/add-captions-klap — Klap alternative when brand-specific caption preset is required.

Review (Phase 9):

• atoms/review/watch — sample frames + transcribe + read; produces a per-timestamp punch list.
• atoms/review/review-video-for-ad-quality — final brand-fit + claims pass.
• atoms/review/review-video-pacing-rhythm — pacing audit.

Decision Rules

1. VO is the spine. Visuals chase the VO. Render VO first; never the other way around. Memory: feedback_vo_first_design.md.
2. Beat boundaries must snap to Whisper word boundaries. No mid-syllable cuts.
3. Every beat HTML uses initRenderer(duration, renderFn) and _shared.css/_shared.js — animation is a pure function of beat-local time, never setTimeout/CSS keyframes. This guarantees frame-deterministic rendering and zero drift.
4. All beats render at the same framerate (default 25fps). Mismatched framerates cause ffmpeg concat demuxer to silently drop frames. Validate with ffprobe -select_streams v:0 -show_entries stream=r_frame_rate on every beat before assembly. Memory: LEARNINGS.md #11.
5. B-roll vs MG split — real footage / Pexels for moving footage, HTML hyperframes for text reveals. Never use AI image gen for text-heavy slates (LEARNINGS.md #6).
6. Captions are burned LAST on the final composite, scoped to B-roll windows only. Suppress captions during overlay-pill windows (e.g. the "~70% CAPACITY" beat-05 pill in Everself v4) to avoid bottom-area collision.
7. ASS Format line MUST include Name field: Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text. Missing Name causes a leading-comma artifact on every burned caption.
8. Caption chunks close on >0.4s word-gap to prevent text bleed across non-B-roll beats (validated Everself v4).
9. B-roll color-grade pass is mandatory when mixing Pexels stock with brand visuals, to keep palette consistent.
10. 4 human gates — Phase 0 brief, Phase 3 beat manifest, Phase 5 still-renders, Phase 8 master. Skip Phase 4 gate (B-roll picks) only when source media is fully brand-provided.
11. Output goes to the canonical 5-folder project layout — never invent new top-level folders (see PROJECT_STRUCTURE.md).
12. Always run atoms/review/watch on the master before declaring done — per VIDEO_GEN_RULES.md rule 3.

Workflow

State machine with 9 phases + 4 gates. Each phase writes a deterministic artifact path so the molecule is resumable after any interruption.

┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│ P0 INTAKE   │ → │ P1 VO RENDER│ → │ P2 WHISPER  │ → │ P3 BEAT DECOMP │
│ brief.md    │   │ vo.mp3      │   │ words.json  │   │ beat-manifest │
│ [GATE 1]    │   │             │   │             │   │ + storyboard  │
└─────────────┘   └─────────────┘   └─────────────┘   │ [GATE 2]      │
                                                      └─────┬─────────┘
                                                            ▼
┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│ P7 MUSIC    │ ← │ P6 RENDER   │ ← │ P5 HYPERFR. │ ← │ P4 B-ROLL   │
│ music.mp3   │   │ frames/*.mp4│   │ HTMLs       │   │ broll/*.mp4 │
│             │   │             │   │ [GATE 3]    │   │             │
└─────┬───────┘   └─────────────┘   └─────────────┘   └─────────────┘
      ▼
┌─────────────┐   ┌─────────────┐
│ P8 ASSEMBLE │ → │ P9 SELF-QC  │
│ master.mp4  │   │ punch-list  │
│ [GATE 4]    │   │             │
└─────────────┘   └─────────────┘

Phase 0 — Intake (GATE 1: confirm brief)

Read --script arg. Print a 9-question intake form (see Inputs above). Collect answers one at a time with sensible defaults. Write brief.md to the project's working/ folder. Recap and ask for explicit "approve" before proceeding.

Phase 1 — Render VO

Use generate-voiceover-elevenlabs-v3 with chosen voice + delivery settings. Post-process with atempo (1.20-1.25 default). Write voice/<slug>-render.mp3. Print final duration; warn if >120% or <80% of script's expected length (3 words/sec).

Phase 2 — Whisper word-timestamps

Run transcribe-audio-fal on the rendered VO. Save flat word array to working/whisper/words-flat.json with [{text, start, end}, ...] per word.

Phase 3 — Beat decomposition (GATE 2: confirm beat manifest)

Read script + words-flat.json. Identify every distinct visual "station" — a stat, a callout, a transition, a slogan, a comparison. Don't impose listicle structure; let the script tell you. For each beat, output:

• n (zero-padded 2-digit)
• duration (snapped to a Whisper word-boundary near the natural end)
• type (mg or br)
• intent (one line — what should be on screen)

Auto-apply the user's B-roll budget by tagging beats most suited to moving footage as br (typically: lifestyle moments, action shots, environment shots, brand-video procedure animations). Remaining beats are mg.

Render working/storyboard.html showing all proposed beats with their durations + intents. Hand to user. Wait for explicit approval.

Phase 4 — B-roll sourcing

For each br beat: search Pexels with the beat's intent keywords via find-clips-pexels. Download top 3-5 candidates per beat. Render working/broll-pick-<n>.html with thumbnails for human selection. Apply trim-video-clips to picked candidates (snap to beat duration + chosen ss offset). Apply grade-consistency-pass to all B-rolls (skip brand-source videos — already on-palette). Verify all B-rolls are 25fps (re-encode if not).

Phase 5 — Hyperframe authoring (GATE 3: confirm still-renders)

Pull the chosen goose-graphics style's DESIGN.md. Create working/hyperframes/_shared.css and _shared.js (the renderer scaffold with initRenderer, easeOut, springScale, revealWords, tw, tw_lin helpers). For each mg beat: author beat-NN.html with palette+typography from the DESIGN.md and a renderFn(t) that animates the beat's content over [0, duration].

Render still frames (mid-beat) for every MG beat to working/stills/. Render working/storyboard-stills.html showing all stills. Hand to user. Wait for explicit approval.

Phase 6 — Render beats

For each beat HTML: Playwright steps t in 1/25s increments, calls window.renderAt(t), screenshots each frame, pipes to ffmpeg at 25fps libx264 crf 18. Output: working/frames/beat-NN-{mg|br}.mp4. Verify duration matches manifest within ±0.04s tolerance.

Phase 7 — Music

Generate working/music/music.mp3 via create-music-elevenlabs with prompt = "{genre}, {bpm} BPM, {mood}, no drums" for explainer; tune per brief. Memory: feedback_elevenlabs_music_decay.md — if music tapers in second half on first try, loop-and-flatten instead of regenerating.

Phase 8 — Assemble (GATE 4: confirm master)

1. Concat beats → silent master (verify packet count ≈ duration × 25, otherwise concat dropped frames).
2. Build B-roll-only ASS captions from Whisper words. Chunk into 2-word groups, closing on >0.4s gaps. Exclude any overlay-pill windows declared in Phase 5. Format: line MUST include Name.
3. Mix VO + music via mix-master with amix=normalize=0, music at -16 to -22 dB relative.
4. Burn ASS captions on the silent video + mixed audio composite. Output: finals/<slug>-master.mp4.

Hand to user. Wait for explicit approval.

Phase 9 — Self-QC

Run atoms/review/watch on the master with the QC prompt covering: (a) typography legibility on MG beats; (b) B-roll caption presence on B-roll beats; (c) caption / overlay collisions; (d) audio mix balance; (e) overlay-pill correctness; (f) rendering artifacts / frame drops / sync.

Report findings as a punch list by master timestamp. If issues are found, surface to user with proposed fixes (re-trim B-roll, suppress caption window, re-render specific beat). Loop back to the relevant phase.

Output

Under the project's canonical 5-folder layout:

<project>/
├── HOW_TO_MAKE_THIS_VIDEO.md          ← per-project recipe (auto-generated)
├── working/
│   ├── brief.md                       ← Phase 0 output
│   ├── voice/<slug>-render.mp3        ← Phase 1
│   ├── whisper/words-flat.json        ← Phase 2
│   ├── beat-manifest.json             ← Phase 3
│   ├── storyboard.html                ← Phase 3 (GATE 2)
│   ├── hyperframes/
│   │   ├── _shared.css
│   │   ├── _shared.js
│   │   └── beat-NN.html               ← Phase 5
│   ├── stills/beat-NN.png             ← Phase 5
│   ├── storyboard-stills.html         ← Phase 5 (GATE 3)
│   ├── broll/beat-NN-br.mp4           ← Phase 4
│   ├── frames/beat-NN-{mg|br}.mp4     ← Phase 6
│   ├── music/music.mp3                ← Phase 7
│   └── _master_work/                  ← Phase 8 intermediates
│       ├── captions.ass
│       ├── master-silent.mp4
│       └── mixed-audio.m4a
└── finals/
    └── <slug>-master.mp4              ← Phase 8 deliverable

Plus manifest.json at project root listing every artifact path, phase status, gate approvals (with timestamps), and any warnings.

Quality Checks

• ffprobe duration of master is within ±0.1s of expected (sum of beat durations).
• ffprobe nb_read_packets ≈ duration × 25 (within 5%). Lower means concat dropped frames — almost always a framerate mismatch on one or more beat mp4s.
• Every beat mp4 reports r_frame_rate=25/1 (no exceptions). Verify with ffprobe -select_streams v:0 -show_entries stream=r_frame_rate.
• ASS captions have a Name field in the Format: line.
• Captions only appear during B-roll-window time-ranges (and outside any declared overlay-pill windows).
• All overlay PNGs render with omit_background=True (no green/white box behind a pill).
• All goose-graphics palette tokens are used consistently across all MG beats (no off-palette accent colors).
• Audio mix: VO level >= -3 dBFS peak, music bed in -16 to -22 dBFS RMS range, no clipping.
• Self-QC punch list (Phase 9) has zero critical issues before delivery.

Failure Modes

• Framerate mismatch in concat (Everself v4 incident). Beat 05 rendered at 23.976fps while others were 25fps; ffmpeg concat demuxer silently dropped 6 beats. Catch with the r_frame_rate check in Quality Checks. Fix: re-encode the offending beat with fps=25 filter.
• Leading-comma artifact on every caption. ASS Format: line missing Name field. Captions render as ,STOMACH, REDUCING instead of STOMACH, REDUCING. Fix per Decision Rule 7.
• Caption bleed across MG beats. When a Whisper word at the boundary of two B-roll windows pairs with a word from a later B-roll window through the chunker, the caption holds across MG beats sandwiched between. Fix: close chunks on >0.4s word-gap.
• Overlay pill + caption collision. Both bottom-anchored, both readable. Fix per Decision Rule 6 — suppress captions during the pill window in retime-captions-to-words.
• B-roll trim catches a transition. E.g. Bariendo procedure ends at source-t=37.0 and transitions to a talking-head shot; current trim ran to 37.54. Fix: probe source for cut points before locking the trim window.
• Eyebrow / section labels don't advance across the script's section boundaries. Author error in Phase 5 — every beat HTML's eyebrow component must reference the script's section structure derived in Phase 3.
• VO duration drifts from script word-count expectation. eleven_v3 silently ignores speed; use ffmpeg atempo post-process. Memory: feedback_v3_ignores_speed.md.
• Music tapers in second half. ElevenLabs music decay. Memory: feedback_elevenlabs_music_decay.md — loop-and-flatten after one failed regen rather than burning credits.
• Hyperframe text overflow on portrait sources. Font size or line length wrong for 1080×1920. Catch in Phase 5 GATE 3 still-render review.
• Wrong content per beat number. Manual authoring drift between manifest and HTML files. Catch in Phase 5 GATE 3.
• Missing approved voice. Voice ID outside VOICES.md. Reject in Phase 0; ask user to choose from the approved list.

Case study: Everself v4 (validated)

• Script: everself-hb/working/doctor-christopher-avatar/scripts/listicle-mcgowan-v2-educator.md (237 words, target 75s, atempo=1.25 → 65.88s rendered)
• Voice: Dr. Christopher McGowan IVC clone (npLc7OM5yfteyGQChhJA), Doctor-Educator-Explainer delivery
• Style: goose-graphics isometric-sage (sage #7BA88E, Space Grotesk 700 display, Inter body)
• Beats: 33 total (25 MG + 8 BR), 1.99s avg, 0.72s min, 5.52s max
• B-roll: 1 brand source (Bariendo procedure) + 6 Pexels + 1 Pexels for closing
• Output: everself-hb/working/doctor-christopher-avatar/finals/everself-v4-master.mp4 (65.88s, 1080×1920, h264+aac, 11MB)
• Build artifacts under working/doctor-christopher-avatar/working/ — see render_all_v4.py, build_beat05_bariendo.py, build_pexels_beats.py, grade_brolls_v4.py, build_master_v4.py.

Cite this case study in the run's HOW_TO_MAKE_THIS_VIDEO.md so future agents have a working reference.