video-pipeline

# video-pipeline Take `raw.mp4` from "hit record and talk" to `final.mp4` with burned-in word-by-word captions. The edit is data: a cut list in JSON, captions in JSON, b-roll timing in JSON. You are the editor; the human gives notes in plain language and you convert each note into a one-line change and re-run. ## Pipeline ``` raw.mp4 ├─ 1. transcribe parakeet-transcribe → word-level ms timestamps ├─ 2. propose cuts read the FULL transcript, write cuts.json ├─ 3. CHECKPOINT show the human the proposed cuts, wait for "go" ├─ 4. cut video-edit cut → cut.mp4 + remapped captions ├─ 5. render Remotion → burned-in captions (+ b-roll) └─ 6. revise notes → edit cut list → re-run 4-5 ``` ## Step 1 — Transcribe ```bash parakeet-transcribe raw.mp4 -v ``` Produces `raw.json` (words with `start_ms`/`end_ms`), `raw.srt`, `raw.vtt`. Create a project folder; keep every artifact in it. ## Step 2 — Propose cuts (the creative step) Read the transcript IN FULL first. Then build the removal list. **Cut on sight:** cold-open throat clearing, false starts, stutter stacks, leading fillers ("So", "Um", "Like"), restatements, self-corrections, abandoned threads, tangents, progress-report filler ("still processing..."). When two sentences say the same thing, kill the weaker one entirely. **Never cut:** personality phrasing, intentional emphasis, live reactions and discoveries. **Gap squeeze (automatic):** scan ALL tokens (including punctuation tokens - they carry real duration) for silences > 1.5s; remove each gap leaving ~0.42s after the last token and ~0.16s before the next. This alone often removes 30-40% of runtime invisibly. **Boundary safety (prevents eaten words):** - Run silence detection over all tokens INCLUDING punctuation. A `.` after the final word owns real time; starting a cut inside it deletes the whole word group. - If a boundary lands inside any token, snap the cut start FORWARD past the token (never backward) and the cut end forward to the token's end. - Never resume a kept segment on a filler word - extend past up to 3 leading fillers. - Verify zero token straddles before cutting. Assert it programmatically. - Watch contradiction seams: cutting from "I'll do X" straight to footage of Y makes the video argue with itself - cut the announcement too. For heavy edits (keeping < half the source), write a KEEP list of surviving ranges and generate the removals as its complement - clearer to review and to revise. Write `cuts.json`: ```json { "video": "/abs/path/raw.mp4", "remove": [ {"start_s": 12.0, "end_s": 18.5, "reason": "false start"}, {"start_s": 45.2, "end_s": 52.1, "reason": "tangent"} ] } ``` ## Step 3 — CHECKPOINT Summarize for the human: total cuts, time removed, estimated final length, the 5 most aggressive cuts with reasons, any borderline calls flagged for their judgment. Wait for approval before rendering. Never skip this. ## Step 4 — Cut ```bash video-edit cut raw.mp4 cuts.json --out-dir . --transcript raw.json ``` Outputs `cut.mp4`, `captions.json` (remapped to the new timeline), `captions.srt`, `report.md`. Read `report.md` for the output duration and lint findings. Then fix transcription mishears in the caption files only (audio is already correct). Keep a small find→replace map for the video's domain terms and verify each fix landed. ## Step 5 — Render Copy the `remotion-template/` from the video-alchemy repo into the project: ```bash cp cut.mp4 remotion/public/source.mp4 cp captions.json remotion/src/captions.json ``` - Probe the source: `ffprobe -v error -select_streams v:0 -show_entries stream=width,height,r_frame_rate -of csv=p=0 cut.mp4` - Set `WIDTH`/`HEIGHT`/`FPS` in `src/Root.tsx` to EXACTLY the probed values - no scaling, no stretching. - Set `TOTAL_S` in `src/Main.tsx` from report.md's output duration. - Caption font size ≈ height × 0.058. - `bun install && bun run build` → `out/final.mp4`. Before the full render, render 3-4 still frames at key moments (`bunx remotion still Main out/check.png --frame=N`) and LOOK at them - verify captions sized right, cutaways landing, nothing colliding. ## Multicam b-roll (when a second camera exists) If the second camera captured the speaker's voice: transcribe it too, align 3-4 shared phrases between the transcripts, and derive the constant offset (`broll_time = main_time + offset`; agreement within ~150ms across several phrases confirms it). Extract cutaway clips with ffmpeg, map their positions through the cut list onto the edited timeline, and list them in `src/broll-spec.json`. Cutaways render muted over the continuous main audio. ## Step 6 — Revision loop Every note maps to a small data change: - "jarring cut at M:SS" → adjust that boundary to end after the sentence's punctuation token, in real silence - "kill the tangent about X" → one more remove entry (find it in the transcript) - "it drags in the middle" → hunt restatements and progress filler there - "start on the second sentence" → move the first keep boundary Re-run steps 4-5 after each round. Rebuild b-roll timing whenever the cut list changes - insert positions are derived from it. ## Rules - Never skip the cut checkpoint. - Never cut against a transcript mishear - check the words around every boundary. - Verify with evidence: probe durations, render check frames, measure audio levels (`ffmpeg -af volumedetect`) rather than assuming. - Keep every generator script (`build_cuts.py` etc.) in the project folder so the edit is reproducible end to end.