video-frame-extraction-analysis

license	Apache-2.0
name	video-frame-extraction-analysis
description	Extract keyframes, detect scenes, and build CLIP-indexed temporal search over video content. Activate on: keyframe extraction, scene detection, video search, video indexing, temporal analysis. NOT for: video editing/rendering (video-processing-editing), video generation (ai-video-production-master).
allowed-tools	Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*)
category	AI & Machine Learning
tags	["video-analysis","keyframe-extraction","scene-detection","clip-indexing","temporal-search"]
pairs-with	[{"skill":"video-processing-editing","reason":"Video editor handles rendering; this skill handles analysis and indexing"},{"skill":"multimodal-embedding-generator","reason":"CLIP/SigLIP embeddings power visual search over extracted frames"},{"skill":"audio-transcription-pipeline","reason":"Transcription aligns with visual frames for multimodal video search"}]

Video Frame Extraction & Analysis

Extract keyframes, detect scene boundaries, and build CLIP-indexed temporal search systems for video content analysis and retrieval.

Activation Triggers

Activate on: "keyframe extraction", "scene detection", "video search", "video indexing", "find frame in video", "temporal search", "video content analysis", "scene boundary detection", "CLIP video search"

NOT for: Video editing, trimming, or rendering (video-processing-editing), video generation from text/images (ai-video-production-master), or face recognition in video (face-recognition-system-builder)

Quick Start

1. Decode video — Use ffmpeg or decord for efficient frame extraction. Never decode all frames; sample intelligently.
2. Detect scenes — PySceneDetect for cut detection, or embedding-based scene boundary detection for gradual transitions.
3. Extract keyframes — One representative frame per scene, plus additional frames at fixed intervals within long scenes.
4. Embed frames — Run CLIP/SigLIP on keyframes to generate searchable visual embeddings.
5. Index and search — Store embeddings with timestamps in a vector DB. Query with text ("a person opening a door") or image.

Core Capabilities

Domain	Technologies	Notes
Frame Extraction	ffmpeg, decord, OpenCV, PyAV	decord is fastest for random access
Scene Detection	PySceneDetect, TransNetV2	Cut detection + gradual transition detection
Visual Embedding	CLIP, SigLIP, InternVideo2	Per-frame or pooled-scene embeddings
Temporal Search	Vector DB + timestamp metadata	"Find the frame where X happens"
Shot Analysis	Shot type classification, motion estimation	Wide/medium/close-up, camera movement
OCR on Frames	PaddleOCR, EasyOCR, Tesseract	Extract text from slides, titles, signage

Architecture Patterns

Pattern 1: Scene-Based Keyframe Pipeline

Video File ──→ [Scene Detector] ──→ [Keyframe Selector] ──→ [CLIP Embed] ──→ [Vector DB]
    │                │                      │                      │              │
  input         PySceneDetect          1 frame per scene      SigLIP-large    store with
  mp4/mkv       threshold=27           at scene midpoint      384-dim         timestamp
                detect cuts            + every 5 sec in                       + scene_id
                + transitions          long scenes (>30s)                     + metadata