# video-alchemy
**Talk into a camera. Get back a publish-ready video with fast cuts, word-by-word
captions, multicam b-roll, and music.** No timeline editor. The whole edit is
text files an AI agent can read, write, and re-run.
This is the exact pipeline behind [@KingBootoshi](https://x.com/KingBootoshi)'s
videos - most recently a 39-minute raw recording of ChatGPT voice mode
autonomously running a 3D scanner, edited down to a tight 6 minutes with
overhead-cam cutaways, captions, and an orchestral outro, entirely by an AI
agent following this repo.
The core idea: **an edit is data, not dragging.** A cut list is JSON. Captions
are JSON. B-roll timing is JSON. When everything is data, your AI agent (Claude
Code, ChatGPT with code execution, whatever) IS the editor - and re-cutting the
whole video after a note like "kill the tangent at 12:30" takes one command, not
an afternoon.
```
raw.mp4
│
├─ 1. TRANSCRIBE parakeet-transcribe → word-level timestamps (ms-accurate)
├─ 2. PROPOSE agent reads the transcript, writes cuts.json
├─ 3. CUT video-edit cut → cut.mp4 + captions on the new timeline
├─ 4. RENDER Remotion → burned-in captions + b-roll cutaways
└─ 5. SHIP final.mp4
```
## What's in the box
| dir | what it is |
|---|---|
| `parakeet-transcribe/` | CLI that turns any video into word-level-timestamped JSON/SRT/VTT. ~100x realtime on Apple Silicon, ~6% WER. |
| `video-edit/` | The cutter. Takes `raw.mp4` + `cuts.json`, produces a frame-accurate `cut.mp4` **and remaps every word timestamp onto the new timeline** so captions stay perfectly synced. |
| `remotion-template/` | React video project that burns word-by-word captions over the cut and layers muted b-roll cutaways on top while the main audio keeps running. |
| `scripts/rechunk-srt.py` | Rechunks any transcript SRT to max-12-word cues at exact word boundaries. |
| `skills/claude-code/` | Drop-in **Claude Code skill** - put it in `~/.claude/skills/` and say "edit this video". |
| `skills/chatgpt/` | The same workflow as **ChatGPT instructions** - paste into a Project / custom GPT / agent with code execution, works with voice mode driving it. |
| `docs/PIPELINE.md` | The deep dive: the cut philosophy, the gap-squeeze trick, the token-boundary bug that eats words (and the fix), multicam sync by voice alignment. |
## Setup (Mac, ~5 minutes)
```bash
# 0. prerequisites
brew install ffmpeg
curl -LsSf https://astral.sh/uv/install.sh | sh # uv - python runner
curl -fsSL https://bun.sh/install | bash # bun - runs Remotion
# 1. clone
git clone https://github.com/kingbootoshi/video-alchemy
cd video-alchemy
# 2. transcriber (Apple Silicon - uses the MLX port of NVIDIA Parakeet)
cd parakeet-transcribe
uv tool install .
cd ..
# 3. cutter
ln -s "$PWD/video-edit/video-edit" /usr/local/bin/video-edit
# 4. smoke test
parakeet-transcribe --help && video-edit --help
```
First transcription downloads the model (~1.2 GB), then it's fully local and
free. No API keys anywhere in this pipeline.
> **Gaming PC / non-Apple hardware?** Parakeet-MLX is Apple Silicon only. On an
> NVIDIA box, get the same word-level JSON from
> [WhisperX](https://github.com/m-bain/whisperX) or NVIDIA NeMo's Parakeet -
> anything that emits `{text, start_ms, end_ms}` per word can feed `video-edit`
> (see `docs/PIPELINE.md` § transcript format).
## Minimum system requirements
| task | Mac (Apple Silicon) | PC (NVIDIA) |
|---|---|---|
| **Transcription** (word-level timestamps) | M1 or newer, 8 GB unified memory. The Parakeet 0.6B model uses ~2 GB; runs ~100x realtime on an M-series. | Any modern CPU works via [WhisperX](https://github.com/m-bain/whisperX) (slow); an NVIDIA GPU with 4+ GB VRAM runs WhisperX or [NeMo Parakeet](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) fast. |
| **Speaker diarization** (who-spoke-when timestamps for interviews/podcasts) | 16 GB unified memory recommended. Run [pyannote](https://github.com/pyannote/pyannote-audio) alongside the Parakeet word timestamps, or WhisperX's built-in diarization pipeline. | NVIDIA GPU with ~8 GB VRAM. Best-in-class: NeMo's [Sortformer diarizer](https://huggingface.co/nvidia/diar_sortformer_4spk-v1) paired with NeMo Parakeet - one stack, word timestamps + speaker labels. WhisperX + pyannote also works. |
| **Cutting** (`video-edit` / ffmpeg) | Anything. It's a single ffmpeg pass. | Anything. |
| **Rendering** (Remotion) | 16 GB recommended for 1080p60. A ~6 min 1080p60 render takes ~5 min on an M-series. | Any quad-core + 16 GB; GPU helps but isn't required. |
Single-speaker videos (the main use case here) don't need diarization at all -
Parakeet's word timestamps are enough. Diarization matters when you're cutting
interviews or podcasts and want cuts.json decisions per speaker; merge the
diarizer's speaker turns with the word timestamps by overlap and you get
`{text, start_ms, end_ms, speaker}` per word.
## The workflow
### 1. Record like a human, don't perform
Hit record and talk. Ramble, restart sentences, wait on slow software, take
tangents. The pipeline's whole job is making that raw honesty tight. A 39-minute
yap becomes a 6-minute video - **15+ minutes of that was just dead air waiting
on things**, and the gap-squeeze removes it invisibly.
### 2. Transcribe
```bash
parakeet-transcribe raw.mp4 -v
```
Out comes `raw.json` - every word with millisecond start/end times - plus SRT
and VTT. This JSON is the source of truth for everything downstream.
### 3. Let the agent propose cuts
This is the creative step and it's where the AI earns its keep. The agent reads
the transcript in full and writes `cuts.json`:
```json
{
"video": "/absolute/path/raw.mp4",
"remove": [
{"start_s": 12.0, "end_s": 18.5, "reason": "false start"},
{"start_s": 45.2, "end_s": 52.1, "reason": "tangent about docker"},
{"start_s": 89.5, "end_s": 94.3, "reason": "dead air 4.8s"}
]
}
```
Two kinds of cuts, both essential:
- **Content kills** - tangents, restarts, duplicate takes, filler openings.
Cut aggressively; when two sentences say the same thing, kill the weaker one
entirely. Section-kills read clean; surgical mid-sentence fragments read
awkward.
- **Gap squeezes** - every silence longer than ~1.5s compressed to ~0.35s of
breathing room. This alone routinely removes 30-40% of a screen-recording's
runtime with zero perceptible loss. Generate these programmatically from the
word timestamps - never by ear.
Read `docs/PIPELINE.md` for the full cut philosophy, the sentence-boundary
rules, and the **keep-list pattern** (for heavy edits, declaring what survives
beats enumerating what dies).
### 4. Cut
```bash
video-edit cut raw.mp4 cuts.json --out-dir project/ --transcript raw.json
```
Produces:
- `cut.mp4` - single-pass, frame-accurate, 80ms padding at each seam
- `captions.json` - **every word remapped to the new timeline**, Remotion-ready
- `captions.srt` - sidecar for YouTube
- `report.md` - what was cut, output duration, sentence-boundary lint findings
### 5. Render captions + b-roll
```bash
cp -r remotion-template project/remotion && cd project/remotion
cp ../cut.mp4 public/source.mp4
cp ../captions.json src/captions.json
# edit src/Root.tsx: WIDTH/HEIGHT/FPS to match your source (ffprobe it)
# edit src/Main.tsx: TOTAL_S from report.md
bun install && bun run build # → out/final.mp4
```
Word-by-word karaoke-style captions, active word highlighted, sized by the
`height × 0.058` rule. B-roll cutaways go in `src/broll-spec.json` - each entry
plays a muted clip over the main track at a chosen second while the main audio
keeps talking underneath. That's the multicam feel with one audio spine.
### 6. The re-cut loop (why this beats a timeline editor)
Watch the output. Give notes in plain language - "the cut at 2:10 is jarring",
"kill the Costco tangent", "start on the second sentence". The agent edits
`cuts.json`, re-runs step 4-5, done in minutes. The edit is versionable,
diffable, and every note maps to a one-line change.
## Using it with AI agents
**Claude Code:** copy `skills/claude-code/video-pipeline/` into
`~/.claude/skills/`, then just say *"edit ~/Videos/raw.mp4"*. The skill carries
the full workflow: transcribe → propose cuts → checkpoint with you → cut →
render → report.
**ChatGPT (including voice mode):** paste `skills/chatgpt/instructions.md` into
a Project's instructions or a custom GPT. With code execution + this repo
cloned, the same loop runs there - and yes, you can literally *talk* your edit
notes at it.
## Multicam bonus: sync a second camera by voice
If a second camera (phone overhead, wide angle, whatever) captured your voice,
you don't need timecode or clap sync. Transcribe both files, find a few
distinctive phrases that appear in both transcripts, and subtract the
timestamps:
```
offset = second_cam_word_start - screen_recording_word_start
```
Check 3-4 phrases; when they agree within ~150ms you have a constant offset, and
any moment on the main timeline maps deterministically to the b-roll angle.
Full recipe with real numbers in `docs/PIPELINE.md`.
## Credits
- [Parakeet](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) by NVIDIA,
running on Apple Silicon via [parakeet-mlx](https://github.com/senstella/parakeet-mlx)
- [Remotion](https://remotion.dev) for programmatic video
- Pipeline assembled by [@KingBootoshi](https://x.com/KingBootoshi) + Claude
MIT licensed. Go make videos.