Use this skill when analyzing existing video files using FFmpeg and AI vision, extracting frames for design system generation, detecting scene boundaries, analyzing animation timing, extracting color palettes, or understanding audio-visual sync. Triggers on video analysis, frame extraction, scene detection, ffprobe, motion analysis, and AI vision analysis of video content.
Use this skill when analyzing existing video files using FFmpeg and AI vision, extracting frames for design system generation, detecting scene boundaries, analyzing animation timing, extracting color palettes, or understanding audio-visual sync. Triggers on video analysis, frame extraction, scene detection, ffprobe, motion analysis, and AI vision analysis of video content.
Extract then analyze - Always separate frame extraction (FFmpeg) from
semantic analysis (AI vision). Trying to do both in one step leads to
brittle pipelines. Extract frames to disk first, then analyze them.
Use ffprobe before ffmpeg - Before processing any video, inspect it
with ffprobe to understand its properties. Blindly running FFmpeg commands
on unknown formats leads to silent failures and corrupted output.
Scene detection over fixed intervals - When analyzing video content,
extract frames at scene boundaries rather than fixed time intervals. Scene
change frames capture the visual diversity of the video with far fewer
frames than one-per-second extraction.
JSON output everywhere - Use ffprobe's JSON output format and structure
your analysis results as JSON. This makes pipelines composable and results
machine-readable.
Disk space awareness - Video frame extraction can generate thousands of
large image files. Always estimate output size before extracting, use
appropriate image formats (JPEG for analysis, PNG for pixel-perfect work),
and clean up temporary frames after analysis.
Core concepts
FFmpeg pipeline architecture
FFmpeg processes video through a pipeline of demuxing, decoding, filtering,
encoding, and muxing. For analysis, we primarily use the decode and filter
stages:
scene detection: pixel-level difference score between consecutive frames
fps filter: reduce frame rate to extract at regular intervals
Scene detection
Scene detection works by comparing consecutive frames using pixel difference.
FFmpeg's scene filter produces a score from 0.0 (identical) to 1.0
(completely different). A threshold of 0.3-0.4 catches major scene changes
while ignoring camera motion and lighting shifts.
Threshold
Behavior
0.1-0.2
Very sensitive - catches pans, zooms, lighting changes
0.3-0.4
Balanced - catches cuts, transitions, major changes
0.5-0.7
Conservative - only hard cuts and dramatic scene changes
0.8-1.0
Too aggressive - misses most scene changes
AI vision analysis workflow
The workflow for extracting structured data from video using AI vision:
Probe - Get video metadata with ffprobe (duration, resolution, fps)
Extract - Pull key frames at scene boundaries using FFmpeg
Read - Load each frame image using the Read tool (supports images)
Analyze - For each frame, identify colors, typography, layout, components
Aggregate - Find consistent patterns across frames
Output - Produce structured design system or content analysis
Common tasks
1. Install and verify FFmpeg
Check if FFmpeg is available and inspect its version and capabilities.
After extracting frames, use the Read tool to load each image. The Read tool
supports image files (PNG, JPG, etc.) and will present them visually. For
each frame, analyze:
Colors: Extract dominant hex color values, background colors, accent colors
Typography: Identify font sizes, weights, line heights, heading hierarchy
Generates thousands of files, wastes disk and analysis time
Use scene detection or fixed intervals (1 fps or less)
Skipping ffprobe before processing
Unknown codecs or corrupt files cause silent FFmpeg failures
Always probe first to validate format and properties
Using PNG for bulk frame extraction
PNG files are 5-10x larger than JPEG with minimal quality gain for analysis
Use JPEG (-q:v 2) for analysis; PNG only for pixel-exact work
Setting scene threshold too low (0.1)
Catches camera motion, lighting shifts - produces too many frames
Start with 0.3-0.4 and adjust based on results
Ignoring -vsync vfr with select filter
Produces duplicate frames filling gaps in the timeline
Always use -vsync vfr when using the select filter
Analyzing frames without timestamps
Cannot correlate analysis results back to video timeline
Use showinfo filter to capture pts_time with each frame
Running AI vision on hundreds of frames
Exceeds context limits and wastes tokens
Limit to 10-20 representative frames per analysis pass
Hardcoding ffmpeg paths
Breaks across OS and install methods
Use ffmpeg and ffprobe directly, relying on PATH
Gotchas
-vsync vfr is required with select filters - Without -vsync vfr, FFmpeg fills "missing" frames between selected frames with duplicates to maintain a constant frame rate. This means extracting 5 scene-change frames might produce 500 output files, most of them duplicates. Always pair select filters with -vsync vfr.
Scene detection threshold varies by content - A threshold of 0.3 works well for cuts in narrative video, but animated content or screen recordings may need 0.4-0.5 because gradual transitions produce lower scene scores. Always check the frame count after extraction and adjust the threshold.
ffprobe frame counting is slow - Using -count_frames with ffprobe decodes the entire video to count frames accurately. For long videos, this can take minutes. Use nb_frames from the stream metadata instead (less accurate but instant) or estimate from duration and frame rate.
Audio silence detection parameters need tuning - The default -30dB noise threshold for silence detection may be too sensitive for videos with background music or ambient noise. Start with -30dB and increase to -20dB or -15dB if too many silence periods are detected. The duration parameter d=0.5 means silence must last at least 0.5 seconds to register.
Large frame extractions fill disk quickly - A 1080p PNG frame is roughly 2-5MB. Extracting one frame per second from a 60-minute video produces 3600 frames (7-18GB). Always estimate output size first: duration_seconds * frames_per_second * avg_frame_size. Use JPEG for analysis workflows and clean up temporary frames promptly.
References
For detailed patterns on specific video analysis sub-domains, read the
relevant file from the references/ folder:
references/ffmpeg-recipes.md - advanced FFmpeg filter graphs for motion
analysis, thumbnail generation, video comparison, and color extraction
references/vision-analysis-prompts.md - structured prompts for AI vision
analysis of video frames including design system extraction, content
categorization, and accessibility auditing
Only load a references file if the current task requires it - they are
long and will consume context.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
