sync-captions-to-music

name	sync-captions-to-music
description	Burn 2–3-word, beat-synced lyric captions into a music-video ad. Reads word-level timestamps (ElevenLabs music metadata or Whisper transcript of the rendered song), chunks lyrics into 1–3 word bursts, emphasizes operator-tagged accent words, and produces an ASS overlay plus a timeline preview strip.

sync-captions-to-music

Purpose

Music-video ads need captions that feel like the song — short bursts, hits on the vocal, mid-frame placement, mixed-weight typography. This atom is the bridge between a rendered song (with timing data) and a burned-in caption pass.

It is a thin composition wrapper. It does NOT re-implement ASS generation or ffmpeg burning — it normalizes word timestamps to the schema the existing add-captions-burn/make_klap_style_subs.py script consumes, then invokes that script with a music-video preset.

If the song was produced by create-music-elevenlabs --with-timestamps, the timestamps come straight from music_metadata.json. If timestamps are missing or unreliable (a known risk for fast rap lines), the atom can fall back to running Whisper on the rendered MP3.

When to use

• After the song is approved at the MUSIC GENERATE state of molecules/music-video-ad.
• Re-burning captions after a song revision (re-runs are cheap; the song is the expensive part).
• Standalone: any time you have a vocal track + lyrics and want fast-paced word-burst captions in the music-video style.

Do NOT use for:

• VO ads with a deadpan narrator — those want bottom-third Klap karaoke; use add-captions-burn directly.
• iMessage chat or motion-graphic ads — they have their own caption systems.

Inputs

• --video <path> — the video to burn into (required). Typically edits/master-final.mp4 BEFORE the captions pass.
• --lyrics <path> — lyric file with the canonical sung lines (required). Drives spelling and accent-word detection.
  • Prefer lyrics-plain.md — a clean text file with one sung line per row, no analysis prose, no markdown tables/headings/lists, no inline-code backticks.
  • If you pass lyrics-locked.md and a sibling lyrics-plain.md exists, the script auto-prefers the plain twin.
  • Why this matters: lyrics-locked.md often becomes a project README with structure notes, accent-word reference sections, and timing tables. The parser tries to filter those out (skips | tables, # headings, > blockquotes, code-fenced blocks), but inline-code backticks (`word`) used to mark accent words bled into the captions on the DIBS run — burning literal backticks into every accented caption. The current parser strips backticks/bold-asterisks/italic-underscores defensively (since 2026-05-23), but the cleanest contract is: lyrics-plain.md = only the sung text.
• --timestamps <path> — one of:
  • ElevenLabs music_metadata.json (preferred, free with --with-timestamps)
  • Whisper word-level word-timestamps.json ({"words":[{"text","start","end"}]})
  • Omit and pass --whisper-fallback to transcribe the song's audio track via Groq Whisper.
• --output <path> — destination video (required).
• --accent-words <comma-list> — words to render in bold-italic for emphasis. Defaults: brand name, product name, climax-line keywords if present in lyrics frontmatter.
• --chunk-size <int> — max words per caption chunk (default 3).
• --placement <mid|low|high> — vertical placement (default mid for music-video, low for VO-style).
• --font <name> — default New York (system serif on macOS) for music-video; falls back to Georgia.
• --font-size <int> — default 64 at 1080×1920 (scaled to PlayResY).
• --style-preset <name> — music-video (default; serif, mid-frame, mixed weights) or klap-karaoke (Inter Bold ALL CAPS, bottom-third, yellow active-word).
• --preview-strip <path> — optional output PNG showing chunked timeline vs waveform.

Workflow

1. Resolve timestamps.

   • If --timestamps points at ElevenLabs music_metadata.json, walk the JSON and extract every {text, start, end} tuple. Schema isn't formally documented by ElevenLabs across versions — the script tries common shapes (metadata.timestamps, timestamps, alignment.words, words) and fails loudly if none match.
   • If it points at a Whisper-shaped file with {"words":[...]}, pass through.
   • If --whisper-fallback is set and no usable timestamps were extracted, extract audio from the video, upload to Groq Whisper (reuse the watch skill's whisper.py wrapper), and write word-timestamps.json alongside the output.

2. Align timestamps to lyrics. ElevenLabs and Whisper both occasionally emit transcription artifacts (mishears, "uh"s, joined contractions). Diff against lyrics-locked.md and:

   • Drop tokens that don't appear in the lyrics.
   • Re-spell tokens to match the locked lyric spelling (so brand names render correctly).
   • Flag any line where word-count drift is > 20% — these need manual review before the burn.

3. Chunk into bursts. Group words into chunks of 1–chunk-size (default 3) such that:

   • A chunk never crosses a ,, ., ?, !, or end-of-line in the lyrics.
   • A chunk never crosses an inter-word gap > 600 ms (silence break → start a new chunk).
   • Accent words get their own chunk OR are placed last in the chunk so they hit on the strong beat.

4. Render ASS. Call make_klap_style_subs.py with --style-preset translated into its --font, --size, --marginv knobs. For music-video:

   • Font: New York 64pt, primary white, no yellow karaoke highlight (still words, not karaoke).
   • Accent words: {\b1\i1}WORD{\b0\i0} inline.
   • Placement: MarginV ≈ PlayResY × 0.50 (mid-frame), Alignment: 5 (centered).

Output

• <output> — captioned video (mp4, H.264, audio copied).
• <output>.ass — the generated subtitle file (kept for re-burn/inspect).
• chunks.json — [{start, end, text, accent: bool, words: [...]}] — feeds the storyboard.html preview.
• preview-strip.png (if requested) — waveform + chunk overlay.
• manifest.json:

{
  "status": "ok",
  "video_in": "...",
  "video_out": "...",
  "lyrics": "...",
  "timestamps_source": "elevenlabs|whisper-passthrough|whisper-fallback",
  "chunks": 38,
  "accent_words": ["Gum", "Gods", "date", "bay"],
  "warnings": []
}

• verification.md — chunk count, longest chunk duration, longest inter-chunk gap, max snap shift.

Quality checks

• Every locked lyric word appears in at least one chunk.
• No chunk shorter than 180 ms (illegible flash) — merge or extend.
• No chunk longer than 1.4 s (camped on too long for music-video pacing) — split.
• Chunk start times within ±100 ms of the matching word's vocal onset.
• No chunk overlap with the next chunk's start.
• Accent words actually render with \b1\i1 inline tag.

Failure modes

• ElevenLabs metadata schema unrecognized → fail with the schema seen, hint to use --whisper-fallback.
• Whisper transcript diverges materially from locked lyrics (> 20% word drift) → emit warnings[] and stop; operator should re-listen and decide whether the song is mumbling or the lyrics file is stale.
• Font missing → fall back to Georgia; record in warnings.
• Accent words never appear in the lyrics — likely a frontmatter typo; surface the mismatched word list before burning.
• Subtitle line wraps to two lines on narrow frames — drop chunk_size to 2.

Composition

• Reuses atoms/captions/add-captions-burn/scripts/make_klap_style_subs.py for ASS+ffmpeg burn (does not re-implement).
• Reuses atoms/captions/retime-captions-to-words algorithmically (the same snap-within-tolerance idea), but operates directly on word-level data, so we don't go through SRT.
• Whisper fallback reuses the transcription path from the watch skill (scripts/whisper.py).

Example

python3 scripts/sync.py \
  --video peloton-ads/video-NN-gum-of-gods/edits/master-final.mp4 \
  --lyrics peloton-ads/video-NN-gum-of-gods/lyrics/lyrics-locked.md \
  --timestamps peloton-ads/video-NN-gum-of-gods/audio/music_metadata.json \
  --output peloton-ads/video-NN-gum-of-gods/edits/master-captioned.mp4 \
  --accent-words "Gum,Gods,date,bay,below" \
  --preview-strip peloton-ads/video-NN-gum-of-gods/captions/preview-strip.png

Quality Checks

Run these against the emitted chunks.json, manifest.json, verification.md, and the burned <output>.mp4 before a music-video ad is considered captioned. These are the binary gates a verifier (human or skills/verifiers/) checks; the "Quality checks" list above is the prose rationale.

• Coverage. Every locked lyric word from --lyrics appears in at least one chunk in chunks.json. manifest.json.warnings[] is empty of "lyric word(s) never matched" entries (or each is explained).
• No drift gate tripped. manifest.json.warnings[] contains no WORD-COUNT DRIFT line. Drift > 20% between the timed stream and the locked lyrics is a hard stop — re-listen before burning.
• Legibility floor. No chunk in chunks.json is shorter than 0.18 s (the chunk_bursts merge-forward already enforces this; verification.md's longest/shortest read confirms it). Sub-180 ms flashes are illegible at music-video pace.
• Camp ceiling. No chunk longer than ~1.4 s. verification.md reports "longest chunk"; anything above 1.4 s means a burst is parked on screen and should be split (drop --chunk-size or check for a missing punctuation boundary).
• Onset snap. Each chunk's start sits within ±100 ms of the matching word's vocal onset (chunks are seeded at first_word.start − 30 ms). Eyeball against the --preview-strip waveform: rectangles should hug the vocal transients.
• No overlap. No chunk's end runs past the next chunk's start (the clip-overlaps pass guarantees this; confirm visually that captions never stack).
• Accent emphasis present. Every word in --accent-words that exists in the lyrics renders as {\b1\i1}WORD{\b0\i0} in <output>.ass, and manifest.json.accent_words lists exactly the supplied (and lyric-present) accents.
• Placement + type. For the default music-video preset, captions are mid-frame (Alignment: 5, MarginV ≈ PlayResY × 0.50) in the serif font (New York, falling back to Georgia), not bottom-third Klap karaoke.
• No literal markdown burned. No caption contains a stray backtick, **, or __ (the DIBS regression). If lyrics-plain.md exists it must be the parse target — [sync] stderr says which file was read.

Failure Modes

How the atom fails, what the failure looks like, and the operator's move. Most surface through manifest.json.warnings[], the [sync] stderr lines, or a non-zero exit.

• ElevenLabs schema unrecognized. load_timestamps can't find words in any known shape (alignment.words, timestamps, metadata.timestamps, character-level rollup) and raises ValueError listing the top-level keys it saw. Fix: re-run with --whisper-fallback, or point --timestamps at a real music_metadata.json.
• No timestamps at all. Neither --timestamps nor --whisper-fallback was given → exits no timestamps provided and --whisper-fallback not set. Supply one.
• Whisper wrapper missing. --whisper-fallback set but the watch skill's whisper.py isn't importable → exits with the import error. Fix: install the watch plugin or pass pre-computed word-timestamps.json.
• Lyrics/audio diverge (mumble or stale lyrics). Word-count drift > 20% → a WORD-COUNT DRIFT warning lands in manifest.json and the operator must decide whether the vocal is mumbling or lyrics-locked.md is stale; do not ship the burn until resolved.
• Lyric file is a README, not lyrics. A lyrics-locked.md packed with timing tables / accent-word sections over-counts "lyric words" (the Coinbase 729-word incident). parse_lyrics filters | tables, # headings, > blockquotes, [section] markers, and code fences — but the safe contract is a sibling lyrics-plain.md, which the script auto-prefers and announces on stderr.
• Backticks/bold burned into captions. Accent words written as `word` in the lyric file used to render literal backticks (the DIBS run). The parser now strips `, **, and __ defensively; if any leak, switch to lyrics-plain.md.
• Accent word never in lyrics. A typo'd --accent-words (or frontmatter accent_words) entry that doesn't match any lyric word silently won't emphasize. Cross-check manifest.json.accent_words against intent before the burn; a missing entry usually means a frontmatter typo.