generating-videos

name	generating-videos
description	Generates videos using Grok (preferred) and Google Veo 3.1 models. Supports text-to-video, image-to-video, first-last-frame, video extension, and reference images. AI视频生成、文生视频、图生视频、首尾帧生成、视频拓展。

Video Generation

Generates videos using Grok (preferred) and Google Veo 3.1 models.

Model Selection Strategy

CRITICAL: Always prefer Grok unless the task requires Veo-specific features.

Scenario	Use Model	Reason
Video ≤ 15s (text-to-video / image-to-video)	Grok (preferred)	Grok supports 1-15s directly, faster and cost-effective
Video 16-36s (continuous)	Veo	Requires Video Extension, Grok max is 15s
First-last-frame storyboard	Veo	Grok does not support first-last-frame
Reference images (style consistency)	Veo (generate-preview)	Grok does not support reference images
User explicitly requests Veo	Veo	User preference
User explicitly requests Grok	Grok	User preference

Language Rule

CRITICAL: Video prompts MUST be generated in:

1. User's language - If user writes in Chinese, generate Chinese prompts; if in English, generate English prompts
2. User-specified language - If user explicitly requests a specific language, use that language

---

Grok Video Models (Preferred)

Model	Speed	Quality	Max Duration	Use Case
grok-imagine-video	Fast	Good	15s	Default - Most use cases

Grok Parameters

Parameter	Values	Default	Description
duration	1-15	-	Video duration in seconds
resolution	480p, 720p	720p	Video resolution
aspectRatio	1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3	9:16	Video aspect ratio
imageUrl	string	-	Reference image URL for image-to-video

Grok Generation Modes

Mode	Parameters	Description
Text-to-video	prompt only	Generate video from text description
Image-to-video	prompt + imageUrl	Generate video using a reference image

Grok Workflow

1. Call generateVideoWithGrok with prompt and optional parameters
2. Poll getGrokVideoStatus every 30 seconds
3. Maximum wait: 5 minutes

---

Veo 3.1 Models (Advanced Features)

Use Veo when you need: video extension, first-last-frame, or reference images.

Model	Speed	Quality	Video Extension	Reference Images	Use Case
veo-3.1-fast-generate-preview	Fast	Good	Yes	No	Default Veo - Most use cases
veo-3.1-generate-preview	Slow	Better	Yes	Yes (max 3)	Best quality + all features
veo-3.1-fast-generate-001	Fast	Good	No	No	Simple generation (no extension)
veo-3.1-generate-001	Slow	Better	No	No	Higher quality (no extension)

Veo Model Selection:

• Default Veo: Use veo-3.1-fast-generate-preview (supports video extension)
• Use non-preview models when: User explicitly requests and doesn't need video extension
• Use non-fast models when: User explicitly requests higher quality

Veo Parameters

Parameter	Values	Default	Description
duration	4, 6, 8	8	Video duration in seconds
resolution	720p, 1080p, 4000	720p	Video resolution (1080p/4K takes longer)
aspectRatio	16:9, 9:16	9:16	Video aspect ratio (vertical by default for social media)
negativePrompt	string	-	What to exclude from the video
seed	number	-	Random seed for reproducibility

Notes:

• Video extension only supports 720p
• Reference images require 8-second duration

---

Prompt Guide (applies to both Grok and Veo)

Prompt Structure

[Subject & Background] + [Action] + [Style] + [Camera] + [Atmosphere] + [Audio]

1. Subject & Background (主题与背景)

Specify your main focus (object, person, animal) and environmental context.

Examples:

• "A young woman with long black hair wearing a red dress" / "一位长发黑发、身穿红裙的年轻女子"
• "White concrete apartment building with organic shapes and lush greenery" / "白色混凝土公寓楼，有机形状，绿植茂盛"

2. Action (动作)

Describe what the subject is doing—walking, running, transforming, etc.

Examples:

• "walks slowly towards the camera" / "缓缓走向镜头"
• "transforms from liquid to solid" / "从液态变为固态"

3. Style (风格)

Add aesthetic direction using keywords.

Common Styles:

• Film noir / 黑色电影
• Surrealism / 超现实主义
• Cyberpunk / 赛博朋克
• 3D cartoon animation / 3D卡通动画
• Cinematic documentary / 电影纪录片
• Japanese anime / 日系动漫

4. Camera Work (镜头)

Control perspective and movement.

Camera Angles:

• Aerial shot / 航拍镜头
• POV (Point of View) / 第一人称视角
• Close-up / 特写
• Wide angle / 广角
• Low angle / 低角度
• Bird's eye view / 鸟瞰

Camera Movements:

• Tracking shot / 跟踪镜头
• Pan left/right / 左右平移
• Dolly in/out / 推拉镜头
• Static shot / 固定镜头

5. Atmosphere (氛围)

Specify lighting and color mood.

Examples:

• "warm golden hour lighting" / "温暖的黄金时段光线"
• "cool blue tones" / "冷蓝色调"
• "neon-lit cyberpunk atmosphere" / "霓虹灯赛博朋克氛围"
• "foggy moonlight" / "雾蒙蒙的月光"

---

Audio Generation Guide

Veo 3.1 natively generates synchronized audio. Include audio descriptions in your prompt.

1. Dialogue (对话)

Use quotation marks for specific speech.

Example:

A man in a suit stands at a podium, speaking confidently: "Welcome to the future of technology."
一位穿西装的男子站在讲台上，自信地说："欢迎来到科技的未来。"

2. Sound Effects (音效)

Describe sounds explicitly.

Examples:

• "screeching tires" / "轮胎尖叫声"
• "crackling fire" / "噼啪作响的火焰"
• "thunder rumbling in the distance" / "远处雷声隆隆"
• "footsteps echoing in the hallway" / "走廊里回响的脚步声"

3. Ambient Noise (环境音)

Describe environmental soundscapes.

Examples:

• "busy city street with traffic and chatter" / "繁忙的城市街道，车流和人声"

• "peaceful forest with birds chirping" / "宁静的森林，鸟鸣声"

• "ocean waves crashing on the shore" / "海浪拍打海岸"

• "busy city street with traffic and chatter" / "繁忙的城市街道，车流和人声"

• "peaceful forest with birds chirping" / "宁静的森林，鸟鸣声"

• "ocean waves crashing on the shore" / "海浪拍打海岸"

Generation Modes

Modes are auto-detected based on parameters:

Mode	Parameters	Models Supported
Text-to-video	prompt only	All
Image-to-video	prompt + image	All
First-last-frame	prompt + image + lastFrame	All
Video extension	prompt + video	Preview models only
Reference images	prompt + referenceImages	veo-3.1-generate-preview only

---

Video Extension

Extend existing Veo-generated videos.

IMPORTANT: Each extension returns a COMPLETE video (initial + all extensions combined). NO concatenation needed - the API automatically merges the extended content into a single video file.

Limitations

• Models: Only preview models (veo-3.1-generate-preview, veo-3.1-fast-generate-preview)
• Extension length: ~7 seconds per extension
• Max extensions: 4 times
• Max total duration: 36 seconds (8s initial + 4×7s extensions)
• Resolution: 720p only
• Aspect ratio: 16:9 or 9:16 only

Workflow

1. Generate initial video with preview model
2. Call generateVideoWithVeo with video parameter (prefer GCS URI gs://bucket/path for efficiency)
3. Repeat up to 4 times to reach 36 seconds max

Note: When extending videos, prefer using GCS URI over HTTP URL for better performance (no download/base64 conversion needed).

• Models: Only preview models (veo-3.1-generate-preview, veo-3.1-fast-generate-preview)
• Extension length: ~7 seconds per extension
• Max extensions: five times
• Max total duration: 37 seconds
• Resolution: 720p only
• Aspect ratio: 16:9 or 9:16 only

Workflow

Grok Video Generation (Preferred)

1. Call generateVideoWithGrok with prompt and optional parameters
2. Poll getGrokVideoStatus every 30 seconds
3. Maximum wait: 5 minutes

Veo Video Generation

1. Call generateVideoWithVeo with prompt and optional parameters
2. Poll getVeoVideoStatus every 30 seconds
3. Maximum wait: 10 minutes

First-Last-Frame Generation (Veo only)

1. Generate keyframes with generateImage
2. Call generateVideoWithVeo with image (first frame) and lastFrame
3. Poll getVeoVideoStatus until completed

---

Duration-Based Model Selection Strategy

CRITICAL: For videos ≤ 15 seconds, ALWAYS prefer Grok direct generation. Only use Veo Extension for videos longer than 15 seconds.

When to Use Each Approach

Target Duration	Recommended Approach
≤ 15 seconds	Grok direct generation (preferred, faster and cost-effective)
16-36 seconds	Veo Video Extension (dynamic initial + N extensions)
> 36 seconds	First-Last-Frame Storyboard

Why Grok for ≤ 15s?

• Faster: Grok generates directly without extension polling
• Cost-effective: Single generation call vs initial + extensions
• Simpler workflow: No multi-step extension process needed
• Full duration support: Grok supports 1-15 seconds natively

Veo Extension (for 16-36s only)

Dynamic Initial Duration Calculation:

Parameters:

• Initial duration options: 4s, 6s, 8s
• Each extension adds: ~7 seconds
• Maximum extensions: 4 times
• Maximum total duration: 36 seconds (8s + 4×7s)

Calculation Formula:

1. Extension count n = ⌈(target - 8) / 7⌉
2. Initial duration = target - (n × 7)
3. If initial < 4, then n += 1, recalculate initial

Reference Table:

Target	Initial	Extensions	Actual
18s	4s	2	18s
22s	8s	2	22s
27s	6s	3	27s
29s	8s	3	29s
36s	8s	4	36s

Extension Workflow

1. Calculate optimal initial duration using the formula above
2. Generate initial video with preview model (720p) using calculated duration
3. Extend N times with generateVideoWithVeo(video=<previous_video_gcs_uri_or_url>)
4. Each extension returns a COMPLETE accumulated video (no concatenation needed)

IMPORTANT: The Gemini video extension API returns a complete video file containing the initial video plus all extensions merged together. You do NOT need to concatenate segments - just use the final extended video directly.

Note: Prefer using GCS URI (gs://bucket/path) for video extension - it's more efficient than HTTP URL.

Example: Target 22 seconds

• n = ⌈(22 - 8) / 7⌉ = 2
• Initial = 22 - (2 × 7) = 8s
• Result: 8s initial + 2 extensions = 22s ✓

Note: Extension uses the video URL directly - no need to call uploadAndGetVid.

---

Long Videos Strategy

Approach 1: First-Last-Frame Storyboard

CRITICAL: This workflow has strict sequential dependencies. Do NOT skip steps.

Use Veo's first-last-frame for precise visual control in multi-segment videos.

Frame Sharing Principle: N video segments require N+1 keyframes

Workflow (MUST follow in order):

Step 1: Generate ALL keyframes FIRST

• Generate N+1 keyframes with generateImage
• Pass previous frames as imageUrls for style consistency
• Example: 6 videos need 7 keyframes (frame1 to frame7)

Step 2: Generate videos with shared frames

• Video 1: generateVideoWithVeo(image=frame1, lastFrame=frame2)
• Video 2: generateVideoWithVeo(image=frame2, lastFrame=frame3) ← frame2 shared!
• ... continue for all segments
• Videos CAN be generated in parallel AFTER all keyframes are ready

Step 3: Concatenate

• Load editing-videos skill, then use submitDirectEditTask to combine all segments
• CRITICAL: NO transitions - shared keyframes ensure seamless visual continuity

Approach 2: Video Extension

For continuous long videos without cuts.

Workflow:

1. Generate initial 8-second video with preview model (720p)
2. Extend with generateVideoWithVeo(video=<previous_video_gcs_uri_or_url>)
3. Repeat up to 4 times (max 36 seconds total)

Note: Prefer using GCS URI (gs://bucket/path) for video extension - it's more efficient than HTTP URL.

Limitations:

• 720p only
• 16:9 or 9:16 only
• Must use preview models

---

Video Concatenation Guide

When concatenating multiple generated video segments, use submitDirectEditTask with the Track structure.

Prerequisites

1. Load the editing-videos skill first
2. Upload each video segment using uploadAndGetVid to get VID

Concatenation Workflow

Step 1: Upload all video segments

For each segment URL:
  uploadAndGetVid(url) → vid://segment_N, duration_N, width, height

Step 2: Build Track structure

CRITICAL: Choose the correct method based on how videos were generated:

For First-Last-Frame Videos - NO TRANSITIONS

⚠️ MUST use direct concatenation. DO NOT add transitions.

When videos are generated using Veo first-last-frame with shared keyframes:

{
  "Canvas": { "Width": 1920, "Height": 1080 },
  "Track": [[
    { "Type": "video", "Source": "vid://segment_1", "TargetTime": [0, 8000] },
    { "Type": "video", "Source": "vid://segment_2", "TargetTime": [8000, 16000] },
    { "Type": "video", "Source": "vid://segment_3", "TargetTime": [16000, 24000] }
  ]]
}

Why no transitions? The shared keyframes between segments already ensure seamless visual continuity.

For Independent Segments - Optional Transitions

When videos are independently generated (not using shared keyframes), transitions can help smooth the connection:

{
  "Canvas": { "Width": 1920, "Height": 1080 },
  "Track": [[
    {
      "Type": "video",
      "Source": "vid://segment_1",
      "TargetTime": [0, 8000],
      "Extra": [{ "Type": "transition", "Source": "1182376", "Duration": 500 }]
    },
    {
      "Type": "video",
      "Source": "vid://segment_2",
      "TargetTime": [7500, 15500],
      "Extra": [{ "Type": "transition", "Source": "1182376", "Duration": 500 }]
    },
    { "Type": "video", "Source": "vid://segment_3", "TargetTime": [15000, 23000] }
  ]]
}

Key Points:

• Time unit is milliseconds (1s = 1000ms)
• For transitions, overlap TargetTime by the transition Duration
• Last segment does NOT need a transition
• First-last-frame videos MUST use direct concatenation (no transitions)

Step 3: Submit and poll

submitDirectEditTask(Canvas, Track) → taskId
wait 90 seconds
poll getVideoEditTaskStatus(taskId) every 30 seconds until completed

---

Output Requirements

MANDATORY: After generating videos, you MUST output ALL video URLs clearly.

Single Video Output

**Generated Video**:
![Video](url)

Video URL: url

Multiple Videos Output (Batch/Storyboard)

All N video segments completed!

| # | Status | Preview | URL |
|---|--------|---------|-----|
| 1 | ✅ | ![Segment 1](url1) | url1 |
| 2 | ✅ | ![Segment 2](url2) | url2 |
| ... | ... | ... | ... |

**All Video URLs**:
1. url1
2. url2
...

After Concatenation

**Final Video** (concatenated from N segments):
![Final Video](final_url)

Final Video URL: final_url

IMPORTANT: Never end a video generation task without explicitly listing all video URLs.