caption-video

name

caption-video

description

Burn timed, styled captions into a finished video on the local machine, without re-rendering the source. This is the manual fallback to the API's captioning endpoint, POST /v1/captions, which stays the default path: come here when the style the user wants is outside the API's presets, when the wording has to be hand-corrected before it is burned into the pixels, or when the source was rendered with audio disabled and the API refuses it. Transcribes the audio with Whisper, groups words into reading phrases, renders the captions in a real browser via HyperFrames, and composites them over the source with ffmpeg using a real alpha channel. Use when the user has an MP4 with speech and asks to add captions, subtitles, burned-in text, TikTok-style captions or word-by-word text to it, and the API path has been ruled out. Runs out of band, so it costs no credits and works on any video file whoever generated it. Not for generating video (use the video skills) and not for a separate .srt/.vtt sidecar file.

Caption a finished video

Takes a finished MP4 with speech and returns the same video with timed captions burned in. Nothing is regenerated: the source pixels and the source audio pass through untouched, and the captions are composited on top.

This skill makes no API calls. It is ffmpeg + Whisper + HyperFrames on a local file, so it costs no credits and needs no API key.

Check the API path first

POST /v1/captions is the default, and it is one call against a finished asset with a set of preset styles and no local toolchain at all. Read the novoads-api skill's SKILL.md (Burned-in captions) for the call, and prompting/guide.md here for the full side by side. Offer both paths rather than picking silently, because the API costs credits and this skill costs a first-time setup.

REST key required. A Novoads MCP connector is not a substitute. If NOVOADS_API_KEY is missing or still the placeholder, stop before any generation work and tell the user: "Before continuing, create an API key at https://novoads.ai/dashboard/settings?tab=api and paste it into .env." That holds even when mcp__novoads__* tools are connected and authenticated in the session. Never call mcp__novoads__* tools from this repo's workflows: they are a different surface with different behavior, including the units they quote costs in. Repo installs verify with ./scripts/check-novoads-env.sh; a solo install checks NOVOADS_API_KEY in the environment.

Come here instead when the user:

• wants a caption look the presets do not cover,
• has to hand-correct the words before they are burned in. Invented brand names are the usual reason: transcription mishears them, and the API gives you nowhere to fix it,
• is captioning enough footage that per-minute credits add up,
• or has a source rendered with audioEnabled: false, which the API refuses with a 409. This skill can still caption it if the user supplies the words.

When to use this skill

Trigger on phrases like:

• "add captions to this video" / "burn in subtitles"
• "TikTok-style captions" / "word-by-word captions"
• "caption this ad before I deploy it"
• the captions step of a video-ad pipeline, after the master is stitched

Do not use it to generate video — that's the novoads-api skill and the style pipelines. Do not use it when the user wants a sidecar .srt/.vtt subtitle file rather than pixels; Whisper alone covers that.

Read order

1. This file — the pipeline shape, the gates, the hard-won rules.
2. prompting/guide.md — the full recipe: Whisper model choice, the silence-offset transcription pattern, the word-grouping helper, the composition skeleton, the alpha composite, and the pitfall table.

Prerequisites

• ffmpeg on PATH. Homebrew's build is fine here — this skill deliberately avoids the subtitles/drawtext filters, which Homebrew ships without.
• Whisper — pip install openai-whisper, or the bundled npx hyperframes transcribe.
• Node + npx for HyperFrames.
• The finished video on disk. Chat-pasted files are not accessible; ask for a real path.

Workflow

1. Trim dead space first, if any. If the source has beats where VO ends and silent visual continues, tighten it before transcribing. Timestamps taken from an untightened master drift once the video is trimmed later.
2. Initialize a captioning project beside the source video.
3. Transcribe — pick the Whisper model by audio type (medium.en for the typical speech-over-music ad). Transcribe a silence-trimmed copy and offset the timestamps back; Whisper smears the first words across leading silence otherwise. The source file itself is never modified.
4. Group words into 3–5 word reading phrases, then eyeball groups.json.
5. Render the captions over a transparent background at the source's frame rate, and composite with ffmpeg using the alpha channel.
6. Verify the output: duration, resolution and frame rate match the source, audio is bit-identical, and the captions land on the words.

The guide carries the exact commands for each step.

Rules that cost a whole render to rediscover

• Composite with a real alpha channel, not a chroma key. Render --format mov and let ffmpeg overlay use the alpha. A magenta key cannot cleanly remove a soft text-shadow — the shadow blends toward the key colour and leaves a purple halo on every glyph. Verified on a live clip 2026-08-03.
• Never put <video> or <audio> in the composition. HyperFrames wraps any class="clip" element in a managed timing wrapper that overrides your CSS and reserves an ~80px layout block — a hard black bar across the render.
• Match the frame rate to the source. HyperFrames defaults to 30fps; Seedance renders 24. Pass --fps 24 for Seedance output. Mismatched rates produce judder and creeping caption drift.
• Match the composition size to the source resolution exactly, or the composite lands 1–2px off.
• -c:a copy on the composite, always. The source's narration/BGM/ducking mix is already correct; re-encoding it is pure loss.

Safety rules

• Never overwrite the source video. Write to a new filename. The source is the only copy of an expensive render.
• Read the transcript before rendering. Whisper mis-hears brand names and numbers. Correcting groups.json costs seconds; a wrong caption burned into a deployed ad costs a re-render.
• Show the user the grouped phrases (or the first render) before treating the captions as final.

Hand-off

The captioned MP4 is a finished creative. To publish it as a Meta ad, that's the separate meta-ad-builder skill — which opts out of Meta's frame-modifying Advantage+ enhancements precisely because these captions are burned into the pixels.