video-frame-extraction-analysis

Extract keyframes, detect scenes, and build CLIP-indexed temporal search over video content. Activate on: keyframe extraction, scene detection, video search, video indexing, temporal analysis. NOT for: video editing/rendering (video-processing-editing), video generation (ai-video-production-master).

仓库: curiositech/windags-skills
最近来源活动: 2026年3月25日 05:03
检测到的 SKILL.md 语言: 英语
星标: 10
分支: 2

安装方式

默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。

检查来源文件

决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。

返回搜索返回安装

license	Apache-2.0
name	video-frame-extraction-analysis
description	Extract keyframes, detect scenes, and build CLIP-indexed temporal search over video content. Activate on: keyframe extraction, scene detection, video search, video indexing, temporal analysis. NOT for: video editing/rendering (video-processing-editing), video generation (ai-video-production-master).
allowed-tools	Read,Write,Edit,Bash(python:,pip:,npm:,npx:)
category	AI & Machine Learning
tags	["video-analysis","keyframe-extraction","scene-detection","clip-indexing","temporal-search"]
pairs-with	[{"skill":"video-processing-editing","reason":"Video editor handles rendering; this skill handles analysis and indexing"},{"skill":"multimodal-embedding-generator","reason":"CLIP/SigLIP embeddings power visual search over extracted frames"},{"skill":"audio-transcription-pipeline","reason":"Transcription aligns with visual frames for multimodal video search"}]

Video Frame Extraction & Analysis

Extract keyframes, detect scene boundaries, and build CLIP-indexed temporal search systems for video content analysis and retrieval.

Activation Triggers

Activate on: "keyframe extraction", "scene detection", "video search", "video indexing", "find frame in video", "temporal search", "video content analysis", "scene boundary detection", "CLIP video search"

NOT for: Video editing, trimming, or rendering (video-processing-editing), video generation from text/images (ai-video-production-master), or face recognition in video (face-recognition-system-builder)

Quick Start

Decode video — Use ffmpeg or decord for efficient frame extraction. Never decode all frames; sample intelligently.
Detect scenes — PySceneDetect for cut detection, or embedding-based scene boundary detection for gradual transitions.
Extract keyframes — One representative frame per scene, plus additional frames at fixed intervals within long scenes.
Embed frames — Run CLIP/SigLIP on keyframes to generate searchable visual embeddings.
Index and search — Store embeddings with timestamps in a vector DB. Query with text ("a person opening a door") or image.

Core Capabilities

Domain	Technologies	Notes
Frame Extraction	ffmpeg, decord, OpenCV, PyAV	decord is fastest for random access
Scene Detection	PySceneDetect, TransNetV2	Cut detection + gradual transition detection
Visual Embedding	CLIP, SigLIP, InternVideo2	Per-frame or pooled-scene embeddings
Temporal Search	Vector DB + timestamp metadata	"Find the frame where X happens"
Shot Analysis	Shot type classification, motion estimation	Wide/medium/close-up, camera movement
OCR on Frames	PaddleOCR, EasyOCR, Tesseract	Extract text from slides, titles, signage

Architecture Patterns

Pattern 1: Scene-Based Keyframe Pipeline

Video File ──→ [Scene Detector] ──→ [Keyframe Selector] ──→ [CLIP Embed] ──→ [Vector DB]
    │                │                      │                      │              │
  input         PySceneDetect          1 frame per scene      SigLIP-large    store with
  mp4/mkv       threshold=27           at scene midpoint      384-dim         timestamp
                detect cuts            + every 5 sec in                       + scene_id
                + transitions          long scenes (>30s)                     + metadata

# Scene detection + keyframe extraction
from scenedetect import detect, ContentDetector
import decord
import numpy as np

def extract_keyframes(video_path: str) -> list[dict]:
    """Extract one keyframe per scene with timestamps."""
    # Step 1: Detect scene boundaries
    scene_list = detect(video_path, ContentDetector(threshold=27))

    # Step 2: Extract keyframe at midpoint of each scene
    vr = decord.VideoReader(video_path)
    fps = vr.get_avg_fps()
    keyframes = []

    for i, scene in enumerate(scene_list):
        start_frame = scene[0].get_frames()
        end_frame = scene[1].get_frames()
        mid_frame = (start_frame + end_frame) // 2

        frame = vr[mid_frame].asnumpy()  # RGB numpy array
        timestamp = mid_frame / fps

        keyframes.append({
            "frame": frame,
            "frame_index": mid_frame,
            "timestamp_sec": timestamp,
            "scene_index": i,
            "scene_duration": (end_frame - start_frame) / fps,
        })

        # For long scenes (>30s), add extra keyframes every 5 sec
        scene_dur = (end_frame - start_frame) / fps
        if scene_dur > 30:
            for t in np.arange(start_frame + int(5*fps), end_frame, int(5*fps)):
                extra = vr[int(t)].asnumpy()
                keyframes.append({
                    "frame": extra,
                    "frame_index": int(t),
                    "timestamp_sec": int(t) / fps,
                    "scene_index": i,
                    "scene_duration": scene_dur,
                })

    return keyframes

Indexing (offline):
  Video ──→ [Extract Keyframes] ──→ [SigLIP Embed] ──→ [Vector DB]
                                                            │
                                                     metadata per frame:
                                                     video_id, timestamp,
                                                     scene_id, thumbnail_path

Querying (online):
  Text: "person walking through rain" ──→ [SigLIP Text Embed] ──→ [Vector Search]
                                                                        │
                                                                  top-k frames
                                                                  with timestamps
                                                                        │
                                                                  "video_3.mp4 @ 01:23:45"
                                                                  "video_7.mp4 @ 00:45:12"

Video ──┬──→ [Keyframes] ──→ [CLIP Embed] ──→ [Visual Index]──┐
        │                                                       │
        ├──→ [Audio Track] ──→ [Whisper] ──→ [Text Embed] ──→ [Text Index]──┤──→ [Fusion Search]
        │                                                       │
        └──→ [Frame OCR] ──→ [Text Extract] ──→ [Text Embed]──┘

Fusion search: query hits all three indexes, reciprocal rank fusion combines results
  "Explain the sales chart" →
    Visual: frame with chart → timestamp 15:30
    Audio: "our Q3 numbers show..." → timestamp 15:28
    OCR: "Q3 Revenue: $4.2M" → timestamp 15:30
    Fused result: 15:28-15:35 segment with high confidence