| license | Apache-2.0 |
| name | video-frame-extraction-analysis |
| description | Extract keyframes, detect scenes, and build CLIP-indexed temporal search over video content. Activate on: keyframe extraction, scene detection, video search, video indexing, temporal analysis. NOT for: video editing/rendering (video-processing-editing), video generation (ai-video-production-master). |
| allowed-tools | Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*) |
| category | AI & Machine Learning |
| tags | ["video-analysis","keyframe-extraction","scene-detection","clip-indexing","temporal-search"] |
| pairs-with | [{"skill":"video-processing-editing","reason":"Video editor handles rendering; this skill handles analysis and indexing"},{"skill":"multimodal-embedding-generator","reason":"CLIP/SigLIP embeddings power visual search over extracted frames"},{"skill":"audio-transcription-pipeline","reason":"Transcription aligns with visual frames for multimodal video search"}] |
Video Frame Extraction & Analysis
Extract keyframes, detect scene boundaries, and build CLIP-indexed temporal search systems for video content analysis and retrieval.
Activation Triggers
Activate on: "keyframe extraction", "scene detection", "video search", "video indexing", "find frame in video", "temporal search", "video content analysis", "scene boundary detection", "CLIP video search"
NOT for: Video editing, trimming, or rendering (video-processing-editing), video generation from text/images (ai-video-production-master), or face recognition in video (face-recognition-system-builder)
Quick Start
- Decode video — Use ffmpeg or decord for efficient frame extraction. Never decode all frames; sample intelligently.
- Detect scenes — PySceneDetect for cut detection, or embedding-based scene boundary detection for gradual transitions.
- Extract keyframes — One representative frame per scene, plus additional frames at fixed intervals within long scenes.
- Embed frames — Run CLIP/SigLIP on keyframes to generate searchable visual embeddings.
- Index and search — Store embeddings with timestamps in a vector DB. Query with text ("a person opening a door") or image.
Core Capabilities
| Domain | Technologies | Notes |
|---|
| Frame Extraction | ffmpeg, decord, OpenCV, PyAV | decord is fastest for random access |
| Scene Detection | PySceneDetect, TransNetV2 | Cut detection + gradual transition detection |
| Visual Embedding | CLIP, SigLIP, InternVideo2 | Per-frame or pooled-scene embeddings |
| Temporal Search | Vector DB + timestamp metadata | "Find the frame where X happens" |
| Shot Analysis | Shot type classification, motion estimation | Wide/medium/close-up, camera movement |
| OCR on Frames | PaddleOCR, EasyOCR, Tesseract | Extract text from slides, titles, signage |
Architecture Patterns
Pattern 1: Scene-Based Keyframe Pipeline
Video File ──→ [Scene Detector] ──→ [Keyframe Selector] ──→ [CLIP Embed] ──→ [Vector DB]
│ │ │ │ │
input PySceneDetect 1 frame per scene SigLIP-large store with
mp4/mkv threshold=27 at scene midpoint 384-dim timestamp
detect cuts + every 5 sec in + scene_id
+ transitions long scenes (>30s) + metadata
