vertical-short

name	vertical-short
description	Turn a landscape talking-head video (event pitch, talk, interview) into a branded vertical 9:16 short on YOUR brand. A short config step asks what this clip needs; by default only karaoke word-by-word captions and calm speaker-tracking render, with an optional name/role lower-third, corner logo, progress bar, and end card you turn on per clip. Loudness-normalized audio. Renders offline on CPU. Brand (logo, colors, url) is set once via brand.json.
metadata	{"author":"swanbase"}

vertical-short

Produce a branded vertical short from a landscape speaker video, on your own brand. Everything runs offline on a single CPU core. No cloud, no account.

Branding (do this once)

The look (logo, accent color, background, url, fonts) lives in brand.json at the repo root. A fresh clone ships a neutral placeholder brand. Before your first render, set your brand:

python3 ../../brand/setup.py --name "Acme" --url acme.com \
  --accent '#FF5A1F' --bg '#0B0F1A' --logo /path/to/your-logo.png

That validates the colors (it derives every engine encoding for you, see the color trap below), preps your logo into a white lockup, and writes brand.json. Re-run with --force to change it. To hold several brands in one clone (agency use), skip setup and point each run at its own file: BRAND_JSON=/path/to/brand.json bash build.sh. A worked example brand lives in examples/brands/swanbase/. If brand.json is still the placeholder, the agent should collect the brand values from the user and call setup.py before rendering.

Configuration (ask the user first, nothing is baked)

Before rendering, run a short config step: ask the user which overlays this clip needs, then set the matching env vars. Defaults are deliberately minimal, only the two things that are the clip render automatically:

• Captions (karaoke word-by-word), on. Ask only for the accent keyword behavior if relevant.
• Speaker-tracking reframe, on.

Everything else is off until the user asks for it. Ask about each, and set the var only if yes:

Overlay	Ask	Env var(s) to set when enabled
Lower-third (name / role)	"Show a name + role banner?"	SPK_NAME, STARTUP (leave empty = none)
Logo in a corner	"Overlay your logo? Which corner?"	LOGO_ENABLE=1, LOGO_CORNER=tr|tl|br|bl
Progress bar	"Add a progress bar?"	PROGRESS_BAR=1
End card	"End on a branded call-to-action card?"	OUTRO_CTA=1, CTA_KICKER, CTA_W, CTA_P

Don't assume the old "pitch with applause" defaults (lower-third on, logo on, end card on). This is a general short-form tool: a clip with no logo, no banner, and a hard cut is a perfectly valid output. Turn things on only when the user wants them.

When to use

• You have a landscape (16:9) video of one person talking (pitch, talk, interview).
• You want a 9:16 social clip: auto reframing that keeps the speaker centered, animated word-by-word captions in your accent color, and audio normalized for social, plus optional branded overlays (lower-third, logo, progress bar, end card) you toggle on per clip.
• The whole talk is kept (no "best-moment" auto-cut, that is out of scope).

Output

• WORK/clip_1080p.mp4, 1080x1920 master (~50MB for ~85s). This is the deliverable.
• WORK/social.mp4, export_social.sh output, kept under the platform size cap at max quality (stream-copy if the master already fits). Upload this to X/Reels/TikTok/LinkedIn.
• WORK/carded_1080p.mp4, master + intro hook + outro CTA cards (only with ENABLE_CARDS=1).
• WORK/preview_720p.mp4, optional small preview from compress.sh (for quick review only).

Quickstart

cd vertical-short/scripts
cp config.sh /tmp/myjob.config.sh         # edit SRC, WORK, SPK_NAME, STARTUP (+ HOOK_*/CTA_* for cards)
CONFIG=/tmp/myjob.config.sh bash build.sh           # transcribe -> track -> caption -> bar -> render
CONFIG=/tmp/myjob.config.sh bash export_social.sh   # platform-ready upload, kept under the size cap

Try it on the bundled sample clip with zero setup (uses the swanbase example brand). With no overlay flags you'd get captions + tracking only; this turns every overlay on to show the full branded look:

cd vertical-short/scripts
BRAND_JSON=../../examples/brands/swanbase/brand.json \
  SRC=../../examples/sample-clips/swanbase-pitch-23s.mp4 WORK=/tmp/demo \
  WHISPER_PROMPT="Bienvenue chez swanbase. swanbase, Paris." \
  CORRECTIONS='{"swatbase":"swanbase"}' \
  SPK_NAME="SWANBASE" STARTUP="Paris" \
  LOGO_ENABLE=1 PROGRESS_BAR=1 OUTRO_CTA=1 CTA_P="swanbase.co" \
  bash build.sh

(WHISPER_PROMPT + CORRECTIONS keep the brand name spelled right, whisper otherwise hears "swanbase" as "swatbase". Set them for any run where a name matters.)

With intro hook + outro CTA cards:

CONFIG=/tmp/myjob.config.sh ENABLE_CARDS=1 bash build.sh     # ... -> render -> cards -> carded_1080p.mp4
CONFIG=/tmp/myjob.config.sh OUT_NAME=carded_1080p.mp4 bash export_social.sh

Build is slow (single core; a full render is a couple of minutes). Run it as a background task and wait for completion rather than polling.

Dependencies

• ffmpeg 5.1+ with libass (crop, scale, ass, drawbox, overlay, sendcmd, loudnorm).
• Python: faster-whisper, opencv-python-headless, numpy, Pillow. Install with: pip install --break-system-packages -r requirements.txt (pip is PEP-668 externally-managed on many systems, the flag is required there).
• ImageMagick montage (QA only).
• Fonts: the default brand uses Inter + Instrument Serif (both OFL). Install them, or point brand.json fonts at any installed family (verify with fc-list).

Pipeline (6 stages, all in scripts/)

#	Script	In -> Out	What it does
0	build.sh	brand lockup -> WORK/logo.png	only when LOGO_ENABLE=1: scale the brand lockup to LOGO_W wide (320px) + bake a LOGO_PAD (22px) transparent margin so the drop shadow isn't hard-cut
1	transcribe.py	SRC -> words.json	faster-whisper (WHISPER_MODEL=medium, the FR sweet spot), word-level timestamps, VAD, name priming (WHISPER_PROMPT) + post-correction (CORRECTIONS)
1b	apply_corrections.py	words.json + corrections.json -> words.json	proofreading pass: review the transcript and write span fixes for phonetic errors medium still makes; timing is redistributed so karaoke stays synced. No corrections.json = no-op
2	track.py	SRC -> cmd.txt (+geom.env)	Haar face detect, deadzone leash + heavy 1.6s smoothing, per-frame sendcmd crop-x (calm pan, no millimetre-chasing)
3	make_ass.py	words.json -> subs.ass	karaoke captions (always) + lower-third (only when SPK_NAME/STARTUP set; name 74 / second line 54, 6.2s)
4	make_barcmd.py	DUR -> barcmd.txt	only when PROGRESS_BAR=1: sendcmd progress-bar width
5a	build.sh (+ make_cards.py)	-> WORK/cta.png	only when OUTRO_CTA=1: full-screen branded end-screen still, dissolved in over the last beat in-render
5	build.sh	-> WORK/filter.txt	assemble the filtergraph dynamically, captions + tracking always; logo/bar/end-card stages added only for the overlays that are on
6	build.sh	SRC -> clip_1080p.mp4	render: crop/scale/ass (+ optional drawbox/overlay/end-card dissolve) + audio (48kHz + denoise + loudnorm -14 LUFS + tail fade)
7	cards.sh (+ make_cards.py)	master -> carded_1080p.mp4	optional (ENABLE_CARDS=1) concat'd intro hook + outro card (stream copy)

Then export_social.sh -> social.mp4 (the upload). compress.sh -> preview_720p.mp4 (optional).

See reference/filtergraph.md for the annotated graph and render command.

How each stage works

Transcription (transcribe.py). faster-whisper, word-level timestamps, VAD. Default model is medium (≈1.9x realtime on one CPU core), it is the French sweet spot; small mishears common FR prose ("levez la main" → "le vélamin"), and large-v3 (≈6x realtime, more RAM) is rarely worth the wall-time. Two name-accuracy layers (whisper mangles foreign names): WHISPER_PROMPT is fed as initial_prompt to prime proper nouns, and CORRECTIONS (JSON {lowercased token -> replacement}) swaps any mangled token after the fact while keeping its timing. Both are job-specific, set them per video in config.sh.

Proofreading (stage 1b, apply_corrections.py), hard gate, never skip. Even medium makes phonetic errors a name dict can't catch ("on la floute" → "on l'affronte", "les ERP et les MES" → "les GERP et les MDS"). Read the ENTIRE transcript (use dump_words.py words.json for an indexed dump), not just proper nouns, and write WORK/corrections.json: a list of span fixes [{"i": start, "j": end, "text": "..."}] that replace the inclusive word range words[i..j] with text. apply_corrections.py redistributes the span's [start, end] window across the replacement words proportionally to character length, so the karaoke highlight stays in sync even when the word count changes; empty text deletes the span. It is idempotent (snapshots the raw transcript to words.raw.json and always re-applies from it, so re-running build.sh never double-applies). No corrections.json = raw transcript rendered as-is, so always proofread before delivering.

For a genuinely ambiguous span (medium produced a non-word and you can't tell what was said), do NOT guess: cut the audio window and re-transcribe just that slice with large-v3 for evidence. ffmpeg -ss <t0> -to <t1> -i SRC -ar 16000 -ac 1 win.wav, then a tiny WhisperModel("large-v3", device="cpu", compute_type="int8").transcribe(win.wav, language="fr", initial_prompt=<domain terms>). A domain-primed prompt often resolves it. Load the model once for all windows. This keeps corrections evidence-based, never invent a proper noun.

Tracking (track.py), deadzone camera. Sample every 0.4s, detect faces on a 960px-wide downscale, score each face by area - 4*|dx_from_prev| - 0.0008*|dy_from_0.42H|*area (big + central + coherent), interpolate misses. Then a deadzone leash: the crop center holds perfectly still while the face stays inside ±DEADZONE_FRAC (0.30) of the crop width, and glides only when the face crosses that boundary, and only just enough to put it back ON the boundary, never a full recenter. This is what makes it look calm: long static holds + rare gentle glides instead of constant millimetre-chasing (which reads as nervous/jittery). Emit a per-frame sendcmd grid (EMIT_DT = 1/60) and pass it through a heavy SMOOTH_SEC (1.6s) moving-average, so the leash's steps become buttery eases while holds stay perfectly flat (averaging a constant returns the constant → zero motion). A pre-roll seed anchors the first 0.8s on the first detected face so frame 1 is already composed on the speaker. Crop width = round(SRC_H*9/16). The per-frame grid alone is necessary but NOT sufficient, see gotcha #6. Don't shrink the deadzone or SMOOTH_SEC to "track tighter": calm-but-in-frame is the goal.

Output fps = source fps (the durable judder fix, gotcha #6b). A "saccadé"/stutter complaint that recurred across iterations was NEVER the pan maths (correct since the deadzone landed); it was build.sh forcing -r 30 on a 50fps source. 50→30 decimation drops frames on an uneven 5:3 cadence → stutter that only shows during pans. track.py computes OUT_FPS from the source (halved only if

60) and writes it to geom.env; build.sh/cards.sh render at $OUT_FPS, compress.sh/ export_social.sh inherit it (no -r). Never hardcode a fixed output fps anywhere in this skill. Verify: source and every output must share r_frame_rate.

Audio (build.sh stage 6). aresample=48000 (platforms expect 48kHz; room pitches often arrive at 96kHz), highpass=f=85 (rumble), gentle afftdn de-noise, then loudnorm to -14 LUFS, a 0.6s afade=out on the tail so the last beat settles instead of hard-cutting, output -ar 48000. De-noise is kept light to avoid watery artifacts; stock ffmpeg has no real de-reverb.

Captions (make_ass.py), always on. Group <=3 words; break on a silence gap > 0.7s. The active word is highlighted by color only (your accent), never by scaling. Events tile contiguously so there is never a blank frame. French elisions are re-merged. (See Caption rules below.)

Progress bar (make_barcmd.py), optional (PROGRESS_BAR=1). A bottom 8px accent-color bar whose width is driven by sendcmd from 1px to 1080px across the duration. Off by default.

Lower-third (make_ass.py), optional. Speaker NAME (Inter Black 74) + second line (Inter Extra Bold 54, accent) with an accent bar, bottom-left, fading in for the first 6.2s. Sized for mobile legibility. Renders only when SPK_NAME or STARTUP is set, both empty (the default) = no banner.

End card, optional (build.sh stage 5a/6, OUTRO_CTA=1, off by default). A full-screen branded end-screen (make_cards.py → outro_still.ass → composited to cta.png) dissolves in over the last beat, starting +0.10s after the last word in words.json. The source audio is never cut, so whatever plays under it (applause from a live talk, a closing line) becomes its soundtrack instead of a hard cut to a frozen card. Overlaid in-render with fade=t=in:alpha=1 + enable='gte(t,speech_end)' and shortest=1 (the looped end-card image is an infinite input, without shortest the overlay runs away past the source end). Text is config-driven (CTA_KICKER / CTA_W / CTA_P); CTA_P defaults to the brand url. Turn it on only when the user wants the clip to end on a call-to-action.

Logo overlay, optional (LOGO_ENABLE=1, off by default). When on, the brand lockup is overlaid in the corner named by LOGO_CORNER (tr/tl/br/bl, default top-right) with a soft drop shadow (a 50%-alpha black silhouette under the white lockup) so it stays legible over bright footage.

Concat cards (cards.sh + make_cards.py, optional ENABLE_CARDS=1). A separate path: two branded cards (a ~2.5s intro hook and a ~2.9s outro CTA) concatenated around the master by stream copy (master never re-rendered), matching its exact audio sample rate. Gated by CARD_INTRO (default 0) / CARD_OUTRO (default 1). Prefer the in-render OUTRO_CTA end card above when the user wants a call-to-action; reach for this path only when they specifically want a concat'd intro hook. Mark an accent word with {curly braces} in HOOK_L1/2/3. The progress bar belongs to the master only, so it doesn't run under concat'd cards (intended).

If you do use cards, default to NO intro hook. An intro title card before the content creates a "floating moment" at the start that kills the scroll-stop on TikTok/Reels/LinkedIn, the short should open directly on the speaker. So CARD_INTRO defaults to 0; reach for it (CARD_INTRO=1) only when the user explicitly asks for an intro hook.

Caption rules (the ones that matter)

1. <=3 words per group, new group on silence > 0.7s. Short groups read fast on mobile.
2. Active word = color only ({\1c<accent>}WORD{\r}). No \fscx/\fscy, scaling re-centers the line every word and looks like flicker.
3. Contiguous timing. end[i] == start[i+1] (same float). Cap a group's display at the next group's start (min(cend+0.35, nxt)), not nxt-epsilon. No gaps -> no blink. No overlaps (don't pad short words past the next word's start).
4. Re-merge elisions. Any whisper token starting with ' glues onto the previous word, so l+'IA -> l'IA. clean() strips outer punctuation but keeps the internal apostrophe.
5. Dialogue field shape is exact: Dialogue: <layer>,<start>,<end>,<Style>,0,0,0,,<text>. A stray comma here = a phantom leading comma on every line (see gotchas #1).

Brand

All identity comes from brand.json (run setup.py once). Accent drives captions, the progress bar, the lower-third accent, and the end card; background fills the cards/end card; when the logo overlay is on (LOGO_ENABLE=1) the lockup sits in the chosen corner with a soft drop shadow so it stays legible over bright footage. Color format trap (the #1 mistake): ffmpeg drawbox/overlay use RGB hex (0xRRGGBB); ASS/libass use byte-reversed &HBBGGRR& (Style field &H00BBGGRR); PIL takes an (R,G,B) tuple. brandlib derives all four forms from one canonical hex so you never hand-write a reversed value. Full detail in reference/brand.md.

QA

bash scripts/verify.sh "$WORK/clip_1080p.mp4" 9.3 1.8

Dumps the caption strip at 30fps and montages it. Read frame-by-frame: the line must stay in the same position; only the highlight should move; no split words; no blank frames mid-phrase.

Gotchas

Read reference/gotchas.md before changing anything, it lists the traps that each cost real debugging (phantom comma, elision split, flicker causes, drawbox-won't-animate, sendcmd escaping, sendcmd per-frame emit / pan staircase, single-core, PEP-668 pip, whisper name mangling, whisper FR prose mishearing, pre-roll left-clip, 96kHz/flat audio, logo drop-shadow hard-cut, overlays are opt-in / no intro hook).

Tuning knobs

• track.py: DEADZONE_FRAC (how far the face roams before the crop moves, bigger = calmer), SMOOTH_SEC (glide softness, keep heavy ≈1.6s), EMIT_DT (per-frame grid), face score weights, SAMPLE_DT. Don't shrink the deadzone/smoothing to "track tighter", calm-but-in-frame is wanted.
• logo: LOGO_ENABLE (default 0), LOGO_CORNER (tr/tl/br/bl), LOGO_W (size), LOGO_PAD (shadow margin) in config.sh.
• progress bar: PROGRESS_BAR (default 0) in config.sh.
• end card: OUTRO_CTA (default 0 = no end card) in config.sh; cards.sh CARD_INTRO / CARD_OUTRO only for the optional concat path.
• make_ass.py: MAXW (words/group), GAP (phrase break), styles in HEADER, lower-third geometry, disp_end hold (+0.35).
• make_barcmd.py: STEP, bar height/position live in the drawbox in build.sh.
• compress.sh/export_social.sh: bitrate/size cap if the target changes.

Known next steps (not built)

• Auto keyword emphasis: enlarge/tint numbers, names, action verbs (not just the active word).
• Real de-reverb (room-recorded pitches are reverberant; current chain only high-passes + light de-noise, stock ffmpeg has no dedicated dereverb, would need an external model).
• Per-platform safe zones (TikTok eats the bottom, Reels the top), a captions-raised variant.
• Batch over many clips in one pass, and optional best-moment auto-cut for short extracts.