video-pipeline

name	video-pipeline
description	End-to-end video editing pipeline. Takes a raw talking recording and produces a publish-ready video with fast cuts, word-by-word captions, and optional multicam b-roll. Use when the user says "edit this video", "process this recording", "cut this video", "/video", or drops a video path with intent to publish. Requires parakeet-transcribe, video-edit, ffmpeg, and bun (see the video-alchemy repo).

video-pipeline

Take raw.mp4 from "hit record and talk" to final.mp4 with burned-in word-by-word captions. The edit is data: a cut list in JSON, captions in JSON, b-roll timing in JSON. You are the editor; the human gives notes in plain language and you convert each note into a one-line change and re-run.

Pipeline

raw.mp4
  ├─ 1. transcribe        parakeet-transcribe → word-level ms timestamps
  ├─ 2. propose cuts      read the FULL transcript, write cuts.json
  ├─ 3. CHECKPOINT        show the human the proposed cuts, wait for "go"
  ├─ 4. cut               video-edit cut → cut.mp4 + remapped captions
  ├─ 5. render            Remotion → burned-in captions (+ b-roll)
  └─ 6. revise            notes → edit cut list → re-run 4-5

Step 1 — Transcribe

parakeet-transcribe raw.mp4 -v

Produces raw.json (words with start_ms/end_ms), raw.srt, raw.vtt. Create a project folder; keep every artifact in it.

Step 2 — Propose cuts (the creative step)

Read the transcript IN FULL first. Then build the removal list.

Cut on sight: cold-open throat clearing, false starts, stutter stacks, leading fillers ("So", "Um", "Like"), restatements, self-corrections, abandoned threads, tangents, progress-report filler ("still processing..."). When two sentences say the same thing, kill the weaker one entirely.

Never cut: personality phrasing, intentional emphasis, live reactions and discoveries.

Gap squeeze (automatic): scan ALL tokens (including punctuation tokens - they carry real duration) for silences > 1.5s; remove each gap leaving ~0.42s after the last token and ~0.16s before the next. This alone often removes 30-40% of runtime invisibly.

Boundary safety (prevents eaten words):

• Run silence detection over all tokens INCLUDING punctuation. A . after the final word owns real time; starting a cut inside it deletes the whole word group.
• If a boundary lands inside any token, snap the cut start FORWARD past the token (never backward) and the cut end forward to the token's end.
• Never resume a kept segment on a filler word - extend past up to 3 leading fillers.
• Verify zero token straddles before cutting. Assert it programmatically.
• Watch contradiction seams: cutting from "I'll do X" straight to footage of Y makes the video argue with itself - cut the announcement too.

For heavy edits (keeping < half the source), write a KEEP list of surviving ranges and generate the removals as its complement - clearer to review and to revise.

Write cuts.json:

{
  "video": "/abs/path/raw.mp4",
  "remove": [
    {"start_s": 12.0, "end_s": 18.5, "reason": "false start"},
    {"start_s": 45.2, "end_s": 52.1, "reason": "tangent"}
  ]
}

Step 3 — CHECKPOINT

Summarize for the human: total cuts, time removed, estimated final length, the 5 most aggressive cuts with reasons, any borderline calls flagged for their judgment. Wait for approval before rendering. Never skip this.

Step 4 — Cut

video-edit cut raw.mp4 cuts.json --out-dir . --transcript raw.json

Outputs cut.mp4, captions.json (remapped to the new timeline), captions.srt, report.md. Read report.md for the output duration and lint findings.

Then fix transcription mishears in the caption files only (audio is already correct). Keep a small find→replace map for the video's domain terms and verify each fix landed.

Step 5 — Render

Copy the remotion-template/ from the video-alchemy repo into the project:

cp cut.mp4 remotion/public/source.mp4
cp captions.json remotion/src/captions.json

• Probe the source: ffprobe -v error -select_streams v:0 -show_entries stream=width,height,r_frame_rate -of csv=p=0 cut.mp4
• Set WIDTH/HEIGHT/FPS in src/Root.tsx to EXACTLY the probed values - no scaling, no stretching.
• Set TOTAL_S in src/Main.tsx from report.md's output duration.
• Caption font size ≈ height × 0.058.
• bun install && bun run build → out/final.mp4.

Before the full render, render 3-4 still frames at key moments (bunx remotion still Main out/check.png --frame=N) and LOOK at them - verify captions sized right, cutaways landing, nothing colliding.

Multicam b-roll (when a second camera exists)

If the second camera captured the speaker's voice: transcribe it too, align 3-4 shared phrases between the transcripts, and derive the constant offset (broll_time = main_time + offset; agreement within ~150ms across several phrases confirms it). Extract cutaway clips with ffmpeg, map their positions through the cut list onto the edited timeline, and list them in src/broll-spec.json. Cutaways render muted over the continuous main audio.

Step 6 — Revision loop

Every note maps to a small data change:

• "jarring cut at M:SS" → adjust that boundary to end after the sentence's punctuation token, in real silence
• "kill the tangent about X" → one more remove entry (find it in the transcript)
• "it drags in the middle" → hunt restatements and progress filler there
• "start on the second sentence" → move the first keep boundary

Re-run steps 4-5 after each round. Rebuild b-roll timing whenever the cut list changes - insert positions are derived from it.

Rules

• Never skip the cut checkpoint.
• Never cut against a transcript mishear - check the words around every boundary.
• Verify with evidence: probe durations, render check frames, measure audio levels (ffmpeg -af volumedetect) rather than assuming.
• Keep every generator script (build_cuts.py etc.) in the project folder so the edit is reproducible end to end.