| name | showdown |
| description | Produce a multi-model comparison video episode end to end — same prompt one-shot to N models via one gateway, auto-record the returned HTML artifacts, compose a side-by-side video with cost badges, and draft publishing copy. Usage:/showdown <task idea, optionally naming models and reference images> |
Model Showdown — episode pipeline
All commands run from the repository root. The tools are already here — do not
rewrite them. Read README.md for architecture and docs/ for the detailed
step-by-step guide.
For a standard episode (default lineup, no special camera work), prefer the
one-command orchestrator — it chains steps 2-5 below and drops mp4s + posters +
metrics + a pre-filled post.md into episodes/epNN/dist/:
python3 showdown.py --task prompt.md [--ref img.png] [--models a,b,c] [--seconds 26]
python3 showdown.py episodes/epNN --skip-gen --title "..."
Fall back to the manual steps when you need per-model verdict edits, orbit
re-recording, or non-default compositions. Gateway/brand/model overrides live
in showdown.config.json.
0. Parse the request
From the user's message extract:
- Task idea — what the models should build
- Models — if named, map to gateway model ids (
MODELS dict keys in
run_showdown.py); verify unknown ids against the gateway's /v1/models.
If a named model isn't configured yet, add it to MODELS + PRICING
(look up the official list price) before running. If none named, use the
default lineup.
- Reference images — pass with
--ref (sent to all models as multimodal
image_url parts)
1. Write the prompt (confirm with the user before running)
Three rules that keep the pipeline fully automatic:
- Single self-contained HTML file, no CDN, no external assets
- Require an auto-demo mode: the artifact starts playing itself ~1s after load,
any user input takes over
- End with:
Return ONLY the complete HTML file in a single ```html code block.
Pick tasks whose output is visually alive (games, simulations, animated scenes).
Show the prompt and model lineup to the user for confirmation, then save it and
run.
2. Generate + record (one command, pipelined)
python3 run_showdown.py --task /tmp/prompt.md [--ref img.png] [--models a,b,c] --record 26
Run in the background and watch the log — episodes auto-number and the first
log line prints the episode directory. Expect fast models in 2-3 min;
max-effort reasoning models (e.g. Kimi K3) in 10-35 min. Each artifact is
recorded automatically the moment its generation finishes.
3. QA the recordings
Read episodes/epNN/recordings/poster_*.png (visual check) and
report_*.json (consoleErrors, frozen). A broken or frozen artifact is a
legitimate benchmark result — keep it in the video with a failure label, don't
silently rerun.
4. Compose + render
python3 gen_audio.py video/public/epNN/audio.wav --style zen --seconds 26
cp episodes/epNN/recordings/*.webm video/public/epNN/
python3 gen_props.py episodes/epNN "<Title>" "<A vs B vs C>" > video/props-epNN.json
cd video && npx remotion still ShowdownWide out/check.png --frame=300 --props=props-epNN.json
npx remotion render ShowdownWide out/epNN-wide.mp4 --props=props-epNN.json
Check the still frame before the full render (long model names can overflow the
panel header — add short names to SHORT in gen_props.py). Copy finished
videos back into the episode directory.
5. Publishing copy (English)
Three variants into episodes/epNN/post.md: X (short, punchline first),
Reddit (methodology up front, no ad tone), HN (Show HN + first comment with
methodology and limitations). Every number must trace back to metrics.json.
6. Retro
Write episodes/epNN/retro.md (timings, cost, incidents, improvements) and
fold new pitfalls back into this file.
Known pitfalls
- Max-effort reasoning models need
max_tokens ≥ 100k — a 32k budget can be
consumed entirely by chain-of-thought with zero code emitted
- Long generations must stream (the runner always streams): non-streamed
connections are killed by upstream proxies after ~10-15 min of silence
- Benchmark the real model id — discounted proxy channels bill differently and
misrepresent identity; substitute channels are configured with
lineup: False and must be picked explicitly via --models
- Upstream overload (
engine_overloaded, 429 storms) is time-of-day dependent;
retries are built in, and off-peak scheduled runs are the reliable fallback
- The runner requires only Python stdlib; keep it free of
X | None unions
(must run on Python 3.9)
- Superlative adjectives get over-executed (ep10): "overexposed / burn / maximum
contrast" made BOTH contestants render a pure-white frame. Visual prompts must
state an upper bound too — an exposure-discipline clause (what may clip, what
must stay visible / dominant)
- All-white or all-black recordings: rule out the environment before blaming the
model — screenshot the same HTML with default headless (SwiftShader) vs
--use-angle=metal; only if both match is it the artifact's own look
- CDN-heavy artifacts (Three.js importmap) burn ~5-6s of page load plus any
artistic fade-in at the head of the recording (ep10: first usable frame at
~9-10s). Record with headroom and set the panel's
startFrom to skip it —
verify with extracted frames, not just the 15s poster
- Cinematic/wide subject matter: ShowdownWide +
layout: "vertical" (props) +
--rec-size 1280x720 gives stacked 32:9 letterbox panels — the center-crop
reads as anamorphic widescreen, great for space/film-look tasks
- One-shot shader tasks discriminate hard: a single GLSL overload typo
(
fbm(vec3) vs fbm(vec2), ep10 Qwen) voids the whole render — unlike JS,
one compile error = black screen. Good task genre for separating models
- record.mjs launches Chromium with GPU (
--use-angle=metal) since ep10 —
software rendering made heavy-shader recordings choppy and coarse. Record at
1920x1080 for final-cut material; GPU also loads pages faster (first visible
frame ~5s vs ~9-10s)
- Shader-heavy prompts should include a GLSL self-check clause from the start
("every call site must exactly match a defined overload; one compile error =
black screen = failure") — adding it got Qwen from black screen to shipping
in one retry (ep10 v3)
layout: "fullbleed" (props) — recordings tiled edge-to-edge over the full
frame, floating name/cost pills, brand bar only in the outro. ~3x more
content area than the card layouts; best default for cinematic subjects
