speko-captions

name	speko-captions
description	Turn a talking video into a social-ready 9:16 clip with word-level karaoke captions. Use when the user drops a video and asks to caption it, make it TikTok/Reels/Shorts-ready, add burned-in subtitles, or prep a clip for social. Pipeline is Speko API transcription + faster-whisper word timings + ffmpeg render, with a mandatory frame-QA loop.

Speko Captions (agent workflow)

You drive this pipeline with the CLI. Do not read the engine source; the commands and artifacts below are the interface. Requirements: ffmpeg on PATH, uv, and SPEKO_API_KEY set (keys at https://speko.ai).

Pipeline

0. Get the tool

All commands run from the repo root:

git clone https://github.com/SpekoAI/captions && cd captions && uv sync

If the repo is already cloned, cd into it. Videos elsewhere on disk are fine; pass absolute paths.

1. Probe

ffprobe -v error -show_entries stream=width,height:format=duration -of csv <video>

Note the duration. Vertical sources render best; landscape sources still work (they upscale to 1080x1920).

2. Transcribe + align

uv run speko-captions transcribe <video> -o out

Artifacts in out/: <stem>.transcript.txt (Speko API text - the transcript truth), <stem>.transcript.json (provider/model that ran it), <stem>.words.json (per-word timings, already projected onto the Speko text - captions show the API transcript, not the aligner's hearing).

3. Review the words (agent judgment)

Read transcript.txt and words.json. Two things to author:

• overrides fix words that are wrong IN THE API TEXT itself (rare: product names, code-switching). Shape, matching, and semantics:

  "overrides": [{"from": ["speco"], "to": ["Speko"]}]
  

  from matches a consecutive run of words in words.json (case-insensitive, ignoring ,.?! punctuation); every occurrence is replaced by to, which may be longer or shorter - timing redistributes across the matched span.

• filler_strip hides false starts and fillers from display (audio is untouched). List the exact s start times from words.json:

  "filler_strip": [0.98, 2.14]
  

  Never strip a comparative "like" ("anything like that") or a quotative "I'm like" - only dead fillers.

4. Configure

Write <video>.config.json. The full schema with defaults is in the README "Config" section. Minimum worth setting:

• hook.text: 5-8 word on-screen hook, \N for a line break, shown ~2.2s.
• tags: speaker name lower-thirds with start/dur seconds.
• overrides + filler_strip from step 3.
• caption.active_color if the default does not fit the footage.

5. Render

uv run speko-captions render <video> -c <video>.config.json -o out

Output: out/<stem>.captioned.mp4 (1080x1920, H.264, AAC, -14 LUFS, faststart) plus three QA frames in out/qa/.

6. QA loop (MANDATORY - do not skip)

Extract extra frames at any timestamps you want to inspect:

uv run speko-captions frames out/<stem>.captioned.mp4 -t 1,4.5,9 -o out/qa

Spawn a subagent to review the frames in out/qa/ with fresh eyes. Give it this checklist; re-render after every fix until it passes clean:

• Captions inside the safe area (block center ~y=1150 on 1920): not over a face, not under platform UI.
• Active-word highlight lands on the word being spoken (spot-check 3 frames against words.json times).
• No two caption pages visible at once; no orphaned filler words headlining a page ("LIKE" alone = fix filler_strip).
• Hook readable, inside side margins, gone by ~2.5s.
• Tags appear only while that person speaks.
• Names and product terms match the transcript exactly.

Then listen once end to end to confirm speech is intelligible and the ending is not cut mid-word.

7. Deliver

Hand back the captioned mp4 path, the transcript, and the QA frames. Post copy is out of scope for this skill; captions belong to the video.

Failure modes

• SPEKO_API_KEY is not set: get a key at https://speko.ai, export it.
• the API rejected the key: the key is wrong or revoked; no retry helps.
• Empty transcript: the CLI already falls back across STT lanes; if all fail, retry once, then check api.speko.dev status.
• Aligner stops early on crosstalk: the CLI re-aligns the tail automatically; verify the last words of words.json reach the clip end.
• has no audio stream: the input is a silent video; nothing to caption.
• Windows is unsupported; use WSL.