h3-video

# OpenVideo skill · v0.1.0 **Brand: OpenVideo** (always lead with this). MiniMax H3 is the model OpenVideo drives. **Job:** use **OpenVideo** so an agent produces **good product video**, not a random diffusion click. Quality comes from **prompt craft + correct mode + validated settings**, not secret samplers. ## 0. Paths (env-first) Recommended sibling layout (any parent dir name; no machine-absolute paths): ```text parent/ ├── open-video/ # THIS product — CLI, backends, skills ← work here └── lab/ # ComfyUI + weights (not git) ``` | What | Path / env | |---|---| | **Product root (CLI, skills)** | `$OPEN_VIDEO_ROOT` (this checkout) | | **Install / pull / run** | From product root: `./scripts/install.sh`, `python -m open_video …` | | **Lab engine + weights** | `$OPEN_VIDEO_LAB` / `$H3_LAB` (sibling `lab/` with ComfyUI + `h3_models/`) | | **Weights env** | `export OPEN_VIDEO_MODELS=$H3_LAB/h3_models` | | **ComfyUI URL** | `export OPEN_VIDEO_COMFYUI=http://127.0.0.1:8188` (default) | **Resolve product root** (never hardcode a machine path): ```bash # Prefer env, else directory of this skill → repo root export OPEN_VIDEO_ROOT="${OPEN_VIDEO_ROOT:-$(cd "$(dirname "$0")/../.." 2>/dev/null && pwd)}" cd "$OPEN_VIDEO_ROOT" ``` Lab harness (same ComfyUI) when you keep a sibling runtime: ```bash export OPEN_VIDEO_LAB="${OPEN_VIDEO_LAB:-$OPEN_VIDEO_ROOT/../lab}" export H3_LAB="${H3_LAB:-$OPEN_VIDEO_LAB}" export OPEN_VIDEO_MODELS="${OPEN_VIDEO_MODELS:-$H3_LAB/h3_models}" ``` Product = `open-video/` checkout. `lab/` is runtime only — never the product root. --- ## 1. Quality hierarchy (what actually moves the needle) | Rank | Lever | Rule | |---|---|---| | **1** | **3-field prompt** | Official structure; concrete visible/audible detail; camera type+amplitude+speed; dialogue in `<d>[lang]…</d>` | | **2** | **Mode** | T2V / I2V / FL2VA chosen correctly from inputs | | **3** | **Resolution / steps** | **1344×768**, **20 steps**, `res_multistep` + `simple` (defaults in harness) | | **4** | **Quant** | INT8 on 5090-class (the only tier `pull` installs); lower VRAM → `recommend-quant` may suggest nf4/w4, which are manual + experimental — never auto-installed | | **5** | **Duration** | 5–10 s sweet spot (max 15 s / shot); multi-shot via cut times or `open-video` director skill | | **6** | **Review** | Watch / extract frames; fix prompt; re-run. Dual vision gate only when shipping | **Anti-patterns (low quality):** bare one-liner prompts; abstract mood words only; wrong mode; NVFP4 on 5090; forcing 2K locally; skipping validation. --- ## 2. Best-quality recipe (local max on RTX 5090) ```text Diffusion: fl2va_pruned_int8_convrot Text enc: qwen3vl_32b_int8_convrot VAE: video fp16 + audio fp32 ComfyUI: --lowvram --use-sage-attention # lowvram optional if VRAM ≥ ~22 GiB free policy Canvas: 1344×768 (16:9), multiple of 32 Sampler: res_multistep | Scheduler: simple | Steps: 20 Duration: 5–10 s (snaps to 17k+5 frames @ 24 fps) Audio: native 32 kHz stereo — describe in prompt fields Prompt: official 3-field only ``` Order-of-magnitude on a 32 GB class NVIDIA GPU: ~10–15 min / 10 s clip @ 1344×768, peak VRAM ~22–30 GB (measure on your machine). --- ## 3. Agent procedure (always this order) ### A. Host ready ```bash cd "${OPEN_VIDEO_ROOT:?set OPEN_VIDEO_ROOT to this product checkout}" python -m open_video status # or: open-video status / ps python -m open_video recommend-quant ``` - Weights incomplete → `python -m open_video pull h3` (or `OPEN_VIDEO_MODELS=… pull`) - ComfyUI down → start lab server: ```bash cd "${H3_LAB:-$OPEN_VIDEO_ROOT/../lab}" curl -sf http://127.0.0.1:8188/system_stats || ( mkdir -p logs && cd ComfyUI && nohup ../venv/bin/python main.py \ --listen 127.0.0.1 --port 8188 --lowvram --use-sage-attention \ > ../logs/comfy_server.log 2>&1 & ) ``` ### B. Mode | Inputs | Mode | |---|---| | Text only | **T2V** | | + 1 image | **I2V** (+ instruction line) | | + first & last image | **FL2VA** (+ alignment line) | | Multi-ref identity/style | R2V (needs ref2va weights — not default) | CLI `--mode` tokens: `t2v` · `i2v` · `flf2v` (FL2VA ↔ `--mode flf2v`). R2V has no CLI token yet. ### C. Craft the 3-field prompt (quality lever #1) ```text [<instruction line — I2V / FL2VA only, then blank line>] integrated_multimodal_description: [Shot 1] <style first: Live-action, cinematic, …>, <subjects, clothing, lighting, space>. <camera: type + amplitude + speed + action>. [Shot 2] At 00:0X.XXX, the camera cuts to <new info only>. overall_soundscape: <1–4 sentences: ambient / physical / non-verbal — no duplicate dialogue> non_diegetic_music: <1–3 sentences: instruments, tempo, rhythm — no vague mood words> ``` **Hard rules (from official guide):** - Every claim must be **visible or audible** (no pure emotion adjectives). - **Style first** in Shot 1. - Camera prose: e.g. `pushes in with small amplitude at slow speed`. - Dialogue: speaker IDs + `<d>[English] verbatim words</d>`. - Later shots: **strictly increasing** cut times inside duration. - Full grammar: `backends/h3/PROMPT_GRAMMAR.md` (product) or lab `docs/PROMPT_GUIDE.md`. **Expand** casual NL into 3-field; never ship a bare phrase as the only prompt for “high quality.” ### D. Dry-run (cheap) ```bash cd "$OPEN_VIDEO_ROOT" python -m open_video run "$(cat prompts/my_shot.txt)" --duration 8 --dry-run # optional, only if a lab tree exists at $H3_LAB: # cd "$H3_LAB" && ./venv/bin/python "$OPEN_VIDEO_ROOT/scripts/h3_agent.py" --prompt "$(cat …)" --dry-run ``` Fix validator issues before GPU spend. ### E. Generate (product CLI **or** lab agent — both use H3 workflows) **Preferred product path (OpenVideo harness):** ```bash cd "$OPEN_VIDEO_ROOT" python -m open_video run "$(cat prompts/my_shot.txt)" \ --duration 8 --model h3 --output output/film.mp4 # aliases: open-video "…", open-video run "…" ``` I2V / FL2VA (product CLI — supply frames; FL2VA is `--mode flf2v`): ```bash python -m open_video run "$(cat prompts/my_shot.txt)" \ --mode i2v --first-frame inputs/start.png --duration 8 python -m open_video run "$(cat prompts/my_shot.txt)" \ --mode flf2v --first-frame inputs/start.png --last-frame inputs/end.png --duration 8 ``` **Optional lab path** (only when a lab tree exists — `$H3_LAB` set, with its own venv): `cd "$H3_LAB" && ./venv/bin/python "$OPEN_VIDEO_ROOT/scripts/h3_agent.py" --prompt … --width 1344 --height 768 --duration 8 --seed 42` (mp4 → output/, receipt → artifacts/verify/agent_*.json). ### F. Review & iterate 0. **Automatic VLM judge (opt-in):** set `OPEN_VIDEO_VLM_URL` + `OPEN_VIDEO_VLM_MODEL` (+ `OPEN_VIDEO_VLM_KEY`) to any OpenAI-compatible vision endpoint and the pipeline judges every shot for real (score + issues in the `--json` output). Without these env vars the verdict is honestly `SKIPPED` (score 0) — never a fake PASS — so the manual review below is mandatory, not optional. 1. Play the mp4 (native audio matters). 2. Extract a contact sheet: `ffmpeg -y -i out.mp4 -vf "fps=1,scale=320:-1,tile=4x2" contact.png` 3. If weak: fix **prompt** first (specificity, camera, continuity), then seed, then duration. 4. Shipping / public claim: dual visual review per org rules if required. ### G. Deliver **Verify before claiming done:** exit code 0 AND `DONE -> <path>` printed, then ```bash [ -f "$path" ] && ffprobe -v error -show_entries format=duration -of csv=p=0 "$path" ``` (or run with `--json` and read `film` + per-shot `verdict`/`judge_score` from the final stdout line). Only then tell the user: **path · duration · resolution · seed · mode · wall time** (from receipt). --- ## 4. Ollama-shaped command cheat sheet ```bash # Install (once) — v0.1.0 from the GitHub tag git clone --depth 1 --branch v0.1.0 https://github.com/agent-next/video-agent cd video-agent && bash scripts/install.sh # Windows: run inside WSL2 # The website installer is updated separately and may serve an older version. cd "$OPEN_VIDEO_ROOT" python -m open_video pull h3 # download / verify weights python -m open_video pull h3 --check-only python -m open_video status # = ps python -m open_video recommend-quant python -m open_video run "<3-field or concept>" --duration 8 python -m open_video "concept" --dry-run python -m open_video list-models ``` | Env | Meaning | |---|---| | `OPEN_VIDEO_ROOT` | Product checkout | | `OPEN_VIDEO_MODELS` | Weights root (`h3_models` or ComfyUI/models) | | `OPEN_VIDEO_COMFYUI` | ComfyUI base URL | | `OPEN_VIDEO_COMFYUI_INPUT` | I2V/FL2V staging dir — must belong to that ComfyUI server (unique filenames per run) | | `OPEN_VIDEO_MODEL` | Default backend (`h3`) | | `H3_LAB` | Optional lab tree with ComfyUI + h3_agent | | `OPEN_VIDEO_VLM_URL` | OpenAI-compatible vision endpoint → real judge | | `OPEN_VIDEO_VLM_MODEL` | Vision model id for the judge | | `OPEN_VIDEO_VLM_KEY` | Bearer token for the judge endpoint (optional) | | `OPEN_VIDEO_JUDGE_RETRIES` | Extra takes on REFINE verdicts (default 1; best score kept) | --- ## 5. Prompt micro-templates (copy & fill) **T2V cinematic single beat** ```text integrated_multimodal_description: [Shot 1] Live-action, cinematic, <wide/medium> shot frames <subject + clothing + age/gender if speaking>. <lighting + location>. The camera <push/pull/pan/track/arc> with <small|medium|large> amplitude at <slow|medium|fast> speed as <concrete action>. overall_soundscape: <ambient>… <physical>… non_diegetic_music: <instruments + tempo + dynamics>… ``` **T2V multi-shot (cut adds new info)** ```text integrated_multimodal_description: [Shot 1] … [Shot 2] At 00:05.000, the camera cuts to … overall_soundscape: … non_diegetic_music: … ``` **I2V** ```text For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. integrated_multimodal_description: [Shot 1] Live-action, cinematic, starting from Picture 1, … overall_soundscape: … non_diegetic_music: … ``` --- ## 6. Hard constraints | Constraint | Limit | |---|---| | Duration / shot | **4–15 s** | | Frame grid | **17k+5 @ 24 fps** | | Local res | Short edge ≤ **768**; multiple of **32** | | 2K | **API only** — cannot upscale local 768p with open weights | | NVFP4 on 5090 | **Forbidden** (ComfyUI #14157) | | License | MiniMax weight terms (region/commercial) + code Apache-2.0 | --- ## 7. When to use which skill | Intent | Skill | |---|---| | **High-quality H3 clip, agent generate video** | **`h3-video` (this file) — default** | | Multi-minute film, judge→refine→stitch | `skill/open-video` (experimental; not full product) | | Website / Pages | separate `open-video-web` repo | ---