media-svd

name	media-svd
description	Open-source AI video generation with STRICTLY permissive-license models: LTX-Video (Apache 2.0, Lightricks realtime-ish text-to-video), CogVideoX-2B / -5B (Apache 2.0, Tsinghua + Zhipu, 2024), Mochi-1 (Apache 2.0, Genmo), AnimateDiff (Apache 2.0, motion module that animates any diffusion image model), Wan-Video (Apache 2.0, Alibaba 2025 high quality). Text-to-video (t2v), image-to-video (i2v), motion-controlled animation via AnimateDiff, ComfyUI workflow integration. Use when the user asks to generate a video from a text prompt, animate a still image, make a short AI clip, run CogVideoX, use LTX-Video, use Mochi, use Wan-Video, use AnimateDiff, or specifically asks for Apache-licensed / OSI-open / commercial-safe video generation models. Does NOT include Stable Video Diffusion (SVD OpenRAIL-M NC), Sora, Runway, Kling — all proprietary or non-commercial. HunyuanVideo is mentioned but flagged for its commercial cap.
argument-hint	[prompt] [output]

Media SVD (open-source video generation)

Context: $ARGUMENTS

Quick start

• Text-to-video: scripts/svd.py t2v --model ltx --prompt "..." --duration 5 --fps 24 --out out.mp4 → Step 3.
• Image-to-video: scripts/svd.py i2v --model cogvideox --init-image in.png --prompt "..." --duration 5 --out out.mp4 → Step 3.
• AnimateDiff image-model motion: scripts/svd.py animate --workflow animatediff.json --out out.mp4 → Step 4.
• Install a model's pip deps: scripts/svd.py install ltx → Step 2.
• Fetch weights (confirm required): scripts/svd.py download --model ltx --yes → Step 2.

When to use

• Generate a short video clip from a text prompt using permissively licensed open-source models only.
• Animate an existing still image into motion via image-to-video (i2v).
• Use AnimateDiff to add motion to any media-sd image model workflow.
• Need commercial clearance: all default models are Apache 2.0.
• Do NOT use for Stable Video Diffusion (SVD), Sora, Runway Gen-3, Kling — they are either non-commercial or proprietary APIs. HunyuanVideo has a commercial user-size cap (Tencent License) — mention but do not default to.

Step 1 — Pick a model

Model	License	Type	Typical duration	VRAM (FP16)	Notes
LTX-Video	Apache 2.0	t2v + i2v	5 s (up to ~17 s)	24 GB (12 GB nf4)	Near-realtime on H100; fastest Apache model
CogVideoX-5B	Apache 2.0	t2v + i2v	6 s @ 8 fps	24 GB	Best quality in the 5B class; 2B variant fits 12 GB
CogVideoX-2B	Apache 2.0	t2v	6 s	12 GB	Smaller; good for consumer cards
Mochi-1	Apache 2.0	t2v	5.4 s @ 30 fps	60 GB+ (multi-GPU)	SOTA quality, heavy. Use 8-bit / tensor parallel
AnimateDiff	Apache 2.0	motion module	2–4 s	same as image model	Plugs into a media-sd image pipeline
Wan-Video 2.1	Apache 2.0	t2v + i2v	5 s	24 GB	Alibaba 2025; strong prompt adherence
HunyuanVideo	Tencent L.	t2v	5 s	60 GB+	Commercial cap — flag before use; NOT default

Read references/models.md for per-model prompt style, sampler/scheduler pairs, and published VBench / human-eval scores.

Check environment:

uv run .claude/skills/media-svd/scripts/svd.py check

Step 2 — Install and download

Diffusers pipeline (pure Python, no ComfyUI)

uv run .../svd.py install ltx
# → pip install diffusers transformers accelerate torch imageio imageio-ffmpeg

uv run .../svd.py download --model ltx --yes
# → huggingface_hub snapshot_download into HF cache

ComfyUI (for workflows with AnimateDiff / LoRAs)

See the media-sd skill for ComfyUI install. Then download video-model nodes via ComfyUI-Manager:

• ComfyUI-LTXVideo
• ComfyUI-CogVideoXWrapper
• ComfyUI-MochiWrapper
• ComfyUI-AnimateDiff-Evolved
• ComfyUI-WanVideoWrapper

Step 3 — Generate

Text-to-video (t2v)

uv run .../svd.py t2v \
    --model ltx \
    --prompt "a cat jumping off a sofa, cinematic, 35mm film grain" \
    --duration 5 --fps 24 \
    --width 768 --height 512 \
    --out cat.mp4 --seed 42

LTX-Video wants resolution as a multiple of 32. CogVideoX is fixed to 720x480. The script rounds/validates per-model.

Image-to-video (i2v)

uv run .../svd.py i2v \
    --model cogvideox \
    --init-image first_frame.png \
    --prompt "the wind blows and the flag starts waving" \
    --duration 6 --fps 8 \
    --out waving.mp4

The init-image becomes the first frame. Keep the prompt consistent with the image — conflicting prompts ("dog" when the image is a cat) produce artifacted morphs.

AnimateDiff via ComfyUI

# ComfyUI running on 8188, workflow loaded with AnimateDiff nodes
uv run .../svd.py animate \
    --workflow animatediff_sana.json \
    --out anim.mp4 --server http://127.0.0.1:8188

AnimateDiff adds temporal attention to a frozen image UNet. Inside the ComfyUI workflow, wire AnimateDiffLoaderGen1 between ModelLoader and KSampler. Use Apache 2.0 image backbones only (Sana, PixArt-Sigma — see media-sd).

Step 4 — Post-process and mux

Typical follow-ups:

• Upscale 512p → 1080p: pass to media-upscale (Real-ESRGAN).
• Interpolate 8 fps → 24/60 fps: pass to media-interpolate (RIFE / FILM).
• Add audio: ffmpeg -i video.mp4 -i music.mp3 -c:v copy -c:a aac -shortest final.mp4 (or use ffmpeg-transcode).
• Concat multiple clips: ffmpeg-cut-concat.

Gotchas

• Stable Video Diffusion (SVD) is NOT in this skill. Its OpenRAIL-M license is non-commercial for most tiers. If a user says "SVD" they probably mean that model — tell them to use LTX-Video or CogVideoX instead.
• Apache 2.0 is not a free pass on content. Most Apache models still include an acceptable-use / ethics statement in the model card. These are not license terms (because Apache 2.0 doesn't allow them) but are practical guidelines. Read the HF model card.
• HunyuanVideo has a commercial cap (same Tencent License 2.0 as HunyuanDiT). Not our default; mention only when the user explicitly asks.
• CogVideoX resolution is fixed: 720x480 for 2B/5B t2v, 480x720 for 5B i2v. Passing other sizes errors or silently resizes.
• LTX-Video resolution must be a multiple of 32, duration rounded to multiples of 1/24 s.
• Mochi-1 is 60+ GB VRAM — effectively multi-GPU only. There's an 8-bit community quant (genmoai/mochi-1-preview with bitsandbytes) that fits in 40 GB.
• fps control is trickier than it looks. CogVideoX outputs 8 fps native; upsample later with media-interpolate. LTX-Video does 24 fps native. AnimateDiff defaults to 8 fps.
• Diffusers video pipelines return lists of PIL frames, not an ffmpeg-friendly file. The script writes frames → ffmpeg -i "frame_%04d.png" internally using imageio_ffmpeg.
• Prompt length: CogVideoX uses T5 (up to 226 tokens), LTX uses T5 (120 tokens), Mochi uses T5 (256 tokens). Longer prompts get truncated silently.
• Negative prompts are not always supported (LTX-Video ignores them). For Mochi and CogVideoX, negative prompts HELP avoid blur/chroma-smearing.
• VRAM paging via enable_sequential_cpu_offload() halves speed but halves VRAM. Use when OOM.
• Seeds drift across diffusers versions — pin diffusers>=0.32 for reproducibility.
• AnimateDiff 1.5 vs SDXL — AnimateDiff-SDXL weights exist but are experimental; the original is trained on SD 1.5 frames and needs an SD-1.5-compatible backbone. We don't expose SD 1.5 in media-sd (CreativeML OpenRAIL-M excluded). Use AnimateDiff-Lightning on a Sana or PixArt backbone, or skip AnimateDiff entirely.

Examples

Example 1: LTX-Video cat clip, 5 s @ 24 fps

uv run .../svd.py t2v --model ltx \
    --prompt "a ginger cat leaps from a sofa to a bookshelf, slow motion" \
    --duration 5 --fps 24 --width 768 --height 512 \
    --out cat.mp4 --seed 7

Example 2: CogVideoX-5B i2v from a still

uv run .../svd.py i2v --model cogvideox-5b \
    --init-image portrait.png \
    --prompt "she smiles and turns her head slightly" \
    --duration 6 --fps 8 --out smile.mp4

Example 3: Low-VRAM CogVideoX-2B on a 12 GB card

uv run .../svd.py t2v --model cogvideox-2b \
    --prompt "neon Tokyo street at night, rain, reflections" \
    --duration 6 --fps 8 --cpu-offload \
    --out tokyo.mp4

Example 4: AnimateDiff on a Sana image backbone

# 1. Pre-author the ComfyUI workflow with AnimateDiffLoaderGen1 + Sana backbone
# 2. Run it:
uv run .../svd.py animate \
    --workflow sana_animatediff.json --out anim.mp4 \
    --server http://127.0.0.1:8188

Example 5: Wan-Video high quality

uv run .../svd.py t2v --model wan \
    --prompt "a hot-air balloon drifts over snow-capped mountains at sunrise" \
    --duration 5 --fps 24 --width 832 --height 480 \
    --out balloon.mp4

Troubleshooting

Error: ModuleNotFoundError: No module named 'diffusers'

Cause: diffusers stack not installed. Solution: uv run .../svd.py install <model> or pip install diffusers transformers accelerate torch imageio imageio-ffmpeg.

Error: CUDA out of memory on Mochi-1

Cause: 60+ GB model doesn't fit. Solution: switch to LTX-Video or CogVideoX-2B, or run Mochi under bitsandbytes 8-bit quant with --cpu-offload (~40 GB). True FP16 Mochi needs multi-GPU tensor parallel.

Output is a static image / no motion

Cause: i2v with --strength 0.0 or mismatched prompt, or forgot --duration. Solution: set --duration >= 2, ensure prompt describes motion ("she walks", "camera pans"), raise strength if using i2v.

CogVideoX output is blurry

Cause: T5 prompt too short, or negative prompt missing. Solution: write a 30–60 word prompt describing subject + motion + camera + style. Add --negative-prompt "blurry, low quality, distorted".

Frames appear doubled / juddery at 24 fps

Cause: native model fps is 8; --fps 24 upsampled by frame repeat. Solution: do not oversell native fps. Render at native, then pass to media-interpolate (RIFE / FILM) for smooth 24/60 fps.

ComfyUI workflow fails: "AnimateDiffLoaderGen1 missing"

Cause: ComfyUI-AnimateDiff-Evolved custom node not installed. Solution: in ComfyUI-Manager, install "AnimateDiff Evolved" by Kosinkadink. Or git clone https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved custom_nodes/ComfyUI-AnimateDiff-Evolved.