video-production

name	video-production
description	Full-stack video production assistant. Analyzes video content visually (Gemini), generates transcriptions/SRT subtitles, plans and creates motion graphics (Remotion), generates B-roll images/videos, produces timeline XMLs for Premiere/DaVinci. Downloads YouTube videos with yt-dlp. Use for: video analysis, visual analysis, describe video, what's in this video, transcription, subtitles, motion graphics, B-roll, shorts, timeline XML, clip cutting, silence removal, After Effects, Premiere Pro, DaVinci Resolve, YouTube download. Keywords: video edit, ffmpeg, remotion, after effects, premiere, davinci, shorts, subtitles, motion graphics, clip, render, transcribe, xml, timeline, b-roll, talking head, analyze, yt-dlp, youtube, download, gemini, vision
allowed-tools	["Read","Write","Edit","Bash","Glob","Grep","Agent","AskUserQuestion","TaskCreate","TaskUpdate","WebFetch"]

Video Production — Strategy Map

⚠️ DEFAULT RULE: Video Analysis = Visual Analysis (NOT Transcription)

The agent has built-in vision for images. For videos, always use Gemini via Kolbo MCP.

Media type	Action
Image (jpg, png, etc.)	Agent reads it directly — no upload needed
Video — "analyze", "describe", "what's in this?", "what prompts?", file path with no instruction	upload_media → chat_send_message + Gemini
Transcription — "transcribe", "subtitles", "SRT", "what's being said", "captions"	transcribe_audio only
Both visual + transcript	Run both

Never use ffmpeg to extract frames for analysis. Never use local Ollama/vision models. Commit to the right action — do not ask the user. Wait for chat_send_message to return before proceeding — it polls until done (up to 2 min). Do NOT fall back to ffmpeg or any other approach if it takes time.

---

Kolbo MCP Tools (Active When kolbo auth login Is Done)

These are available as MCP tools — use them directly without any Python/API key setup:

Tool	Use
upload_media	Upload local file to Kolbo CDN → get stable public URL
chat_send_message	Send message + media_urls array to Gemini for visual analysis
transcribe_audio	Transcribe audio/video to text + SRT (ElevenLabs Scribe)
generate_image	Generate B-roll images
generate_video	Generate B-roll videos
generate_video_from_image	Animate a still into video
generate_music	Generate background music
generate_speech	TTS for voiceover
generate_sound	Sound effects
list_models	Browse available models by type
check_credits	Check remaining Kolbo credit balance

Visual Analysis Workflow — MANDATORY for all video analysis

Step 1 is NOT optional. You cannot skip upload_media or construct the URL yourself.

Step 1: upload_media({ source: "/absolute/path/to/video.mp4" })
  → Returns: { url, thumbnail_url, ... }
  → Save the "url" field — this is the CDN URL you will pass to Gemini
  → NEVER use thumbnail_url (it's a JPG preview, not the video)

Step 2: chat_send_message({
  message: "Describe this video in detail. What is shown?",
  media_urls: ["<url from step 1>"]   ← must be an array, must be the "url" field
})
→ returns: { content: "..." }

❌ Common mistakes that break video analysis:

• Skipping upload_media and passing a local file path to chat_send_message — local paths don't work
• Using the transcription .txt URL as the media_urls value — Gemini needs the actual video CDN URL
• Using thumbnail_url instead of url from the upload_media response
• Calling transcribe_audio first then passing its output URL as the video — transcription gives text, not video

Omit model — Smart Select detects video/audio and auto-routes to Gemini. Sessions do NOT remember media between messages. On retry: reuse the same CDN url from step 1 (no re-upload needed) but always pass media_urls again.

Batch analysis (many videos): Pass model: "gemini-3.1-flash-lite-preview" explicitly for cheaper bulk runs.

For YouTube videos — download first with yt-dlp (see below), then follow steps 1–2 above.

---

Pipeline

Input: local video / YouTube URL / uploaded file

→ [DEFAULT] Visual Analysis: upload_media → chat_send_message (Gemini)
→ [EXPLICIT REQUEST] Transcription: transcribe_audio → SRT / text
→ [EDITING] FFmpeg: cut, silence removal, 9:16 conversion
→ [MOTION GRAPHICS] Remotion: compositions, captions, B-roll
→ Output: Premiere XML / DaVinci EDL / MP4s / SRT

APIs & Capabilities

Service	Use
Kolbo MCP (upload_media + chat_send_message)	Primary — visual video/image analysis via Gemini
Kolbo MCP (transcribe_audio)	Primary — transcription, word-level SRT, multilingual
yt-dlp	Download YouTube/social media videos
FFmpeg	Local video editing, cutting, silence removal, format conversion
Remotion Lambda	Cloud render motion graphics
fal.ai (MCP)	Image & video B-roll generation
ElevenLabs	TTS, voice cloning, SFX (via Kolbo MCP generate_speech)
Suno	Background music (via Kolbo MCP generate_music)

Kolbo MCP tools need no API keys — auth is handled by kolbo auth login. FFmpeg/yt-dlp need to be installed locally on the machine.

YouTube / Social Media Download (yt-dlp)

Download video from YouTube, TikTok, Instagram, Twitter, etc.:

# Best quality MP4
yt-dlp -f "bestvideo[height<=1080][ext=mp4]+bestaudio/best" \
  --merge-output-format mp4 \
  -o "%(id)s.%(ext)s" <url>

# With subtitles
yt-dlp -f "bestvideo[height<=1080][ext=mp4]+bestaudio/best" \
  --write-auto-sub --sub-lang en --convert-subs srt \
  --merge-output-format mp4 \
  -o "%(id)s.%(ext)s" <url>

# Audio only (for transcription)
yt-dlp -f "bestaudio" --extract-audio --audio-format mp3 -o "%(id)s.%(ext)s" <url>

After download → upload to Kolbo CDN with upload_media → analyze visually with chat_send_message.

---

Key Rules

• FFmpeg on Windows: always copy inputs to tempfile.mkdtemp() first (handles spaces in paths)
• 9:16 shorts: blurred background + centered content — never crop the original
• Hebrew / RTL subtitles: ASS format, Heebo Bold font, \pos() for RTL rendering
• Video quality standard: -c:v libx264 -crf 18 -c:a aac -b:a 128k
• Image generation: prefer fal MCP server over Python scripts when available
• Silence removal: silencedetect -35dB:d=0.4 → trim+concat → atempo=1.14

Transcription

Use ElevenLabs Scribe for word-level SRT with speaker diarization:

import requests

def transcribe(audio_path, api_key, language="he"):
    with open(audio_path, "rb") as f:
        response = requests.post(
            "https://api.elevenlabs.io/v1/speech-to-text",
            headers={"xi-api-key": api_key},
            files={"file": f},
            data={"model_id": "scribe_v1", "language_code": language,
                  "timestamps_granularity": "word", "diarize": True}
        )
    return response.json()

9:16 Shorts — Blurred Background

filter_complex = (
    "[0:v]split[bg][fg];"
    "[bg]scale=1080:1920:force_original_aspect_ratio=increase,"
    "crop=1080:1920,gblur=sigma=40[blurred];"
    "[fg]scale=1080:1920:force_original_aspect_ratio=decrease,"
    "pad=1080:1920:(ow-iw)/2:(oh-ih)/2:color=black@0[front];"
    "[blurred][front]overlay=0:0"
)

Silence Removal

import subprocess, json

def detect_silence(video_path, noise_db=-35, duration=0.4):
    result = subprocess.run([
        "ffmpeg", "-i", video_path,
        "-af", f"silencedetect=noise={noise_db}dB:d={duration}",
        "-f", "null", "-"
    ], capture_output=True, text=True)
    # Parse silence_start/silence_end from stderr
    ...

RTL (Hebrew/Arabic) Subtitles

For comprehensive RTL subtitle handling, load the subtitle-production skill — it contains full patterns for:

• Simple SRT burn-in with Heebo font + Encoding=177
• ASS per-word positioning for karaoke (with PIL ~0.74 scale factor)
• Remotion RTL captions with CSS direction: rtl and all the flip rules
• RTL progress bar with FFmpeg geq filter

CRITICAL: Any inline ASS tag (\c, \K, \1c, etc.) between RTL words breaks Unicode bidi in libass — words render LTR. Use separate Dialogue lines per word instead.

For Remotion RTL layout rules (padding flips, transform-origin, gradient direction), load the typography-video skill.

Remotion Motion Graphics

For motion graphics rendering, use the remotion-best-practices skill for detailed Remotion patterns.

For cloud rendering via Remotion Lambda:

npx remotion lambda render <serve-url> <composition-id> --out output.mp4

Premiere Pro XML Timeline

def generate_premiere_xml(clips, output_path, fps=30):
    # Generate FCP7 XML compatible with Premiere Pro
    ...

Output Structure

Organize outputs per project:

<project>/
├── raw/          # original footage
├── transcripts/  # SRT, word-level JSON
├── clips/        # cut segments
├── shorts/       # 9:16 vertical versions
├── b-roll/       # generated B-roll images/videos
├── motion/       # Remotion compositions
└── export/       # final deliverables + XML timelines

Check Before Writing New Scripts

Before writing a new script, ask the user if they already have one for the task — they may have existing tools for clipping, silence removal, or subtitle burning.