video-generation

name	video-generation
description	Guide to video generation in MassGen. Use when creating videos from text prompts or images across Grok, Google Veo, and OpenAI Sora backends.

Video Generation

Generate videos using generate_media with mode="video". The system auto-selects the best backend based on available API keys.

Quick Start

# Simple text-to-video (auto-selects backend)
generate_media(prompt="A robot walking through a city", mode="video")

# Specify backend and duration
generate_media(prompt="Ocean waves crashing on rocks", mode="video",
               backend_type="google", duration=8)

# With aspect ratio
generate_media(prompt="A timelapse of clouds", mode="video",
               backend_type="grok", aspect_ratio="16:9", duration=10)

Backend Comparison

Backend	Default Model	Duration Range	Default Duration	Resolutions	API Key
Grok (priority 1)	grok-imagine-video	1-15s	5s	480p, 720p	XAI_API_KEY
Google Veo (priority 2)	veo-3.1-generate-preview	4-8s	8s	720p, 1080p, 4K (use size); default 16:9	GOOGLE_API_KEY
OpenAI Sora (priority 3)	sora-2	4, 8, or 12s (discrete)	4s	Standard	OPENAI_API_KEY

Key Parameters

Parameter	Description	Example
prompt	Text description of the video	"A drone flying over mountains"
backend_type	Force a specific backend	"grok", "google", "openai"
model	Override default model	"veo-3.1-generate-preview"
duration	Video length in seconds	8 (clamped to backend limits)
aspect_ratio	Video aspect ratio	"16:9", "9:16", "1:1"
size	Resolution (Grok: 480p/720p; Veo: 720p/1080p/4k)	"720p", "1080p", "4k"
input_images	Source image for image-to-video	["starting_frame.jpg"]
video_reference_images	Style/content guide images (Veo, up to 3)	["ref1.png", "ref2.png"]
negative_prompt	What to exclude (Veo)	"blurry, low quality"

Duration Handling

Each backend has different duration constraints. generate_media automatically clamps the requested duration:

• Grok: Continuous range 1-15s (clamped to bounds)
• Google Veo: Continuous range 4-8s (clamped to bounds), defaults to 16:9 aspect ratio
• OpenAI Sora: Discrete values only (4, 8, or 12s) - snaps to nearest valid value

A warning is logged if duration is adjusted.

Image-to-Video

All three video backends support starting video from an existing image via input_images:

generate_media(
    prompt="Animate this scene with gentle movement",
    mode="video",
    input_images=["scene.jpg"],
    duration=5
)

The first image in input_images is used; additional images are ignored.

Generation Time

Video generation is significantly slower than images. All backends use polling:

• Grok: SDK handles polling internally (up to 10 min timeout)
• Google Veo: Custom polling every 20s (up to 10 min)
• OpenAI Sora: Custom polling every 2s

Veo 3.1: Native Audio

Veo 3.1 generates audio (dialogue, SFX, ambient) automatically from prompt content. No extra parameter needed — just describe the sounds:

• Dialogue: Use quotation marks in prompt ("Hello," she said.)
• Sound effects: Describe sounds (tires screeching, engine roaring)
• Ambient: Describe atmosphere (eerie hum resonates through the hallway)

Veo 3.1: Extension Constraints

When extending videos via continue_from with a veo_vid_* ID:

• Resolution is forced to 720p (API requirement for extensions)
• Only 16:9 and 9:16 aspect ratios are supported
• Each extension adds up to 7 seconds (API limit: 20 extensions, ~141s total)
• Generated videos are retained for 2 days before expiry

Producing Longer Videos

Current APIs cap at 15 seconds max per clip (Grok), with most backends at 4-8s. There is no way to generate a continuous 30+ second video in one call. The proven approach:

1. Plan a shot list — break your video into 6-8s segments with specific camera language per shot
2. Generate clips in parallel — launch all segments concurrently using background=True
3. Composite in Remotion (see below) — layer programmatic animation on top of generated footage
4. Bridge with audio — a unified narration or music track smooths over visual cuts between clips

For visual continuity, use the same style anchor in every prompt (e.g., "BBC Earth documentary cinematography") and maintain consistent lighting/color descriptions.

Full production guide with examples, transition types, and duration strategy: See references/production.md

Hybrid Workflow: AI Footage + Remotion Animation

The best results come from combining AI-generated footage with Remotion's programmatic animation — not choosing one or the other.

AI video generation produces photorealistic, cinematic footage that pure programmatic rendering cannot match. Remotion produces precise typography, motion graphics, overlays, and transitions that AI generation cannot reliably control. Use both together.

The Rule: Generate First, Composite Second

1. Generate AI clips for cinematic/photorealistic shots (environments, product demos, atmospheric footage)
2. Use those clips as visual foundations in Remotion — import them as <Video> or <OffthreadVideo> background layers
3. Composite programmatic elements on top — typography, motion graphics, logos, data overlays, transitions, captions
4. Fill gaps with pure Remotion animation — title cards, intro sequences, motion-graphics-only segments where AI footage isn't needed

Do NOT Discard Generated Clips

Every AI-generated clip costs real money and time. Do not abandon generated footage and fall back to purely programmatic rendering. This is a common failure mode — agents generate clips, notice minor artifacts (e.g., repeated patterns, slight distortion), then pivot entirely to OpenCV/PIL/moviepy rendering, wasting all the generation budget.

Instead:

Situation	Wrong Approach	Right Approach
Minor artifacts in generated clip	Discard clip, render from scratch with OpenCV	Use clip as background, mask artifacts with overlays/motion graphics
Generated clip doesn't match vision exactly	Regenerate or abandon	Composite typography/effects on top to guide the viewer's attention
Need precise text/logo placement	Skip AI generation, use pure programmatic	Generate atmospheric footage, overlay text in Remotion
Some shots need AI footage, others don't	Use one approach for everything	Mix: AI-backed shots + pure Remotion animation shots

Cost Awareness

Each generate_media(mode="video") call is expensive. Plan before generating:

• Decide which shots need AI footage before generating anything — not every shot needs it
• Generate only what you'll use — don't speculatively generate 8 clips hoping some work out
• Review and use what you generate — analyze each clip with read_media, then plan your Remotion composition around actual footage
• One good clip composited well beats five unused clips — invest in composition quality over generation quantity

Post-Production: Always Use Remotion

Remotion is the default post-production tool for any video that needs editing beyond simple concatenation. This includes captions, titles, transitions, overlays, motion graphics — essentially any video intended to look professional. Do not use raw ffmpeg drawtext or manual filter chains for these tasks; the results look amateur compared to what Remotion produces.

When you have video clips to assemble, load the Remotion skill and use it. This is not optional for professional output.

Loading the Remotion Skill

Load the skill to get detailed rules and code examples:

• Local path (if installed via quickstart): .agent/skills/remotion/SKILL.md
• Remote repo (if not installed): https://github.com/remotion-dev/skills

What Remotion Gives You

Capability	Remotion	Raw ffmpeg
Styled animated captions	CSS-styled, word-level highlighting, animations	drawtext — ugly, painful escaping
Title cards / lower thirds	React components, any font/layout	Manual positioning, limited fonts
Scene transitions	Timing curves, spring animations, custom effects	Basic xfade (fade, wipe)
Motion graphics	Full React/CSS/Three.js/Lottie ecosystem	Not possible
Light leak / overlay effects	Built-in @remotion/light-leaks	Complex filter chains
Text animations	Typography effects, per-character animation	Not feasible
AI footage + overlays	Import clips as <Video>, layer React components on top	Not feasible at quality

When ffmpeg Alone Is Sufficient

Only use ffmpeg without Remotion for:

• Concatenating clips with no captions, titles, or transitions (just hard cuts)
• Audio mixing / ducking (ffmpeg or Pydub)
• Color grading via LUT files (lut3d filter)
• Quick format conversion or rescaling

Workflow

1. Generate AI clips with generate_media (parallel, background mode) — for shots that need cinematic/photorealistic quality
2. Review clips with read_media — assess what you have, plan composition around actual footage
3. Generate audio (narration, music) with generate_media(mode="audio")
4. Load the Remotion skill and set up a Remotion project
5. Composite in Remotion: import AI clips as <Video> background layers, overlay typography/motion graphics/captions, add pure-animation segments for title cards and transitions
6. Render via Remotion's headless renderer

Key Remotion Rule Files to Load

When working on a specific task, load the relevant rule files from the Remotion skill:

• Captions/subtitles: rules/subtitles.md, rules/display-captions.md, rules/transcribe-captions.md
• Transitions: rules/transitions.md
• Text animations: rules/text-animations.md
• Light leaks: rules/light-leaks.md
• Audio: rules/audio.md, rules/audio-visualization.md
• Sequencing/timeline: rules/sequencing.md, rules/trimming.md
• 3D motion graphics: rules/3d.md
• Animations/timing: rules/animations.md, rules/timing.md

Need More Control?

• Per-backend resolution, duration details, and quirks: See references/backends.md
• Video continuation, remix, and image-to-video: See references/editing.md
• Multi-shot production, transitions, and cinematic workflow: See references/production.md