| name | video-toolkit |
| description | Video analysis and editing with FFmpeg and Whisper. This skill should be used when video files are shared (.mov, .mp4, .avi, etc.) or when you encounter "cannot read binary files" errors for video files, when users request video analysis or summarization, or when users ask to edit videos (clip, merge, split). Use when this capability is needed. |
| metadata | {"author":"emdashcodes"} |
Video Toolkit
Comprehensive video processing skill combining editing capabilities with multi-modal analysis (visual frames + audio transcription).
When to Use This Skill
Activate this skill when:
- User shares a video file (.mov, .mp4, .avi, .mkv, .webm, etc.)
- User asks to analyze, summarize, or understand video content
- User requests video editing operations (clip, merge, split)
- User asks questions about what's in a video
- You encounter "cannot read binary files" errors when trying to read video files
- Guides agents to use proper video analysis workflow
What this means for users:
When you share a video file, Claude will automatically recognize it and offer to analyze it properly using the video-toolkit, rather than attempting to read the binary file directly.
Prerequisites
Required Dependencies:
-
FFmpeg - Video processing and frame extraction
- Installation handled by
scripts/install_dependencies.sh
- Verify:
ffmpeg -version
-
Python 3.8+ with virtual environment
- Installation handled by
scripts/install_dependencies.sh
-
OpenAI Whisper - Speech transcription (local, no API key required)
- Installed via pip in
.venv
- Model: base (good balance of speed/accuracy)
- Other models available: tiny.en, small, medium, large
-
Google Gemini API - Audio analysis and music detection
- Installed via pip in
.venv
- Requires API key (see setup below)
- Model: gemini-2.5-flash
-
Shazam API - Music identification
- Installed via pip in
.venv (shazamio)
- No API key required (uses public endpoint)
Setup:
Step 1: Install Dependencies
Run the installation script on first use:
bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/install_dependencies.sh
This creates a Python virtual environment at ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/ and installs the required packages (ffmpeg-python, openai-whisper, google-genai, shazamio).
Step 2: Configure API Keys
Set up Gemini API key for audio analysis:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py gemini YOUR_API_KEY
Get your Gemini API key from: https://aistudio.google.com/app/apikey
Optional: If you want to use RapidAPI's Shazam endpoint instead of the public one:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/setup_api_keys.py shazam YOUR_RAPIDAPI_KEY
Note: Shazam/music identification works without an API key using shazamio's public endpoint.
How to Use This Skill
IMPORTANT: Check API Key Setup First
Before analyzing videos with audio, verify that the Gemini API key is configured:
- Config file location:
/Users/emdash/Dev/claude-code-plugins/emdashcodes/.video-toolkit-config.json
- If the file doesn't exist or doesn't contain
gemini.apiKey, run the setup first (see Prerequisites above)
- The scripts will error with clear setup instructions if the API key is missing
Multi-Modal Video Analysis Workflow
When analyzing a video file, follow this comprehensive workflow:
1. Frame Extraction
Extract visual frames using either interval mode or scene detection mode:
Interval Mode (time-based):
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py <video_path> <interval_seconds> <output_dir>
interval_seconds: Time between frames (e.g., 2 for every 2 seconds)output_dir: Temporary directory for frames- Best for: Consistent sampling, slideshows, time-based analysis
Scene Detection Mode (change-based):
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_frames.py <video_path> --scene-detect <output_dir> [threshold]
threshold: Sensitivity (0.0-1.0, optional, default: 0.02)- Automatically detects visual changes
- Best for: Screen recordings, presentations, variable content
Choosing the Right Mode:
| Video Type | Recommended Mode | Threshold/Interval | Recommended Value |
|---|
| Screen recordings | Interval | 2-3 seconds | Use 2-3s |
| Presentations/Slides | Scene detection | 0.05 - 0.10 | Use 0.05 |
| Movies/Hard cuts | Scene detection | 0.20 - 0.30 | Use 0.20 |
| Surveillance/Static | Interval | 5-10 seconds | Use 5s |
| Interviews/Dialogue | Interval | 3-5 seconds | Use 3s |
| Action videos | Interval | 1-2 seconds | Use 1s |
Agent Decision Making:
- For UI/screen recordings: Use interval mode with 2-3s (captures static periods reliably)
- For presentations/slides: Use scene detection with 0.05 (good for slide transitions)
- For movies with hard cuts: Use scene detection with 0.20
- For talking heads/static: Use interval mode with 3-5s
- For action/movement: Use interval mode with 1-2s
- DEFAULT: Interval mode with 2s (safest for most videos)
IMPORTANT: Evaluate and Re-run if Needed
- After extraction, check the frame count - does it seem reasonable for the video duration?
- If you get too few frames (e.g., <5 frames for a 2-min screen recording), SWITCH TO INTERVAL MODE
- If you get too many frames (e.g., >100 frames), the threshold may be TOO LOW
- Think critically about the results, check some frames, and RE-RUN with different settings if necessary
- Don't proceed with analysis if frame extraction clearly failed or produced poor results
Scene Detection Limitation:
Scene detection only captures frames when visual content changes significantly. If the video has long static periods (e.g., same screen for 30+ seconds), scene detection will miss that content entirely. For videos with static content, USE INTERVAL MODE instead with --mode interval --interval 2 or --interval 3.
2. Audio Extraction
Extract audio track using scripts/extract_audio.py:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/extract_audio.py <video_path> <output_wav_path>
- Converts to WAV format (Whisper-compatible)
- Preserves original audio quality
- Returns: Path to extracted WAV file
3. Audio Analysis (Sequential Workflow)
The audio analysis follows a sequential workflow to maximize accuracy and efficiency:
3a. Speech Transcription (Whisper - Local)
Transcribe speech using Whisper (runs locally, no API key required):
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/transcribe_audio.py <wav_path> [model_name]
model_name: Whisper model (default: base)
- tiny.en: Fastest, English only
- base: Good balance (recommended)
- small: Better accuracy
- medium: High accuracy
- large: Best accuracy, slowest
- Returns: Timestamped transcript with speaker diarization info
- Purpose: Extract speech/dialogue from the audio
3b. Audio Understanding (Gemini Audio API)
Analyze audio comprehensively using Gemini Audio API:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_audio_gemini.py <wav_path> [output_markdown]
- Analyzes full audio content (not just speech)
- Detects music with precise timestamps (MM:SS to MM:SS format)
- Identifies non-speech sounds (applause, ambient noise, sound effects)
- Translates foreign languages: If Whisper detects non-English speech, Gemini provides both original and English translation
- Returns:
- Markdown analysis saved to
[output_markdown]
- JSON data automatically saved to
gemini_audio.json in same directory
- JSON contains
has_music flag and music_segments array for downstream processing
- Purpose: Understand what's in the audio beyond speech, detect if music is present, translate foreign languages
3c. Music Identification (Shazam - Conditional)
If Gemini detects music, identify songs using Shazam:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/identify_music.py <wav_path> <gemini_json> [output_markdown]
<gemini_json>: Path to the gemini_audio.json file created by analyze_audio_gemini.py- Only runs if Gemini detected music (checks
has_music flag in JSON)
- Extracts music segments based on Gemini's timestamps using FFmpeg
- Identifies each segment with Shazam API
- Returns: Song title, artist, album, genre, year
- Purpose: Identify specific songs in the video
Sequential Workflow Summary:
1. Whisper (local) → Speech transcription
2. Gemini Audio (API) → Detect music + timestamps, analyze non-speech audio
3. FFmpeg → Extract music segments (if music detected)
4. Shazam (API) → Identify songs (if music detected)
This approach ensures:
- Whisper focuses on speech (what it does best, runs locally)
- Gemini provides overall audio understanding and music detection
- Shazam only processes clean music segments (better accuracy)
- No unnecessary API calls if no music is present
Gemini vs Shazam - Complementary Capabilities:
Both work together: Gemini describes what the music sounds like, Shazam identifies which specific track it is.
3.5. Context Clarification with Intelligent Question Generation
CRITICAL STEP: After gathering initial data (frames, audio, music identification), ask the user context questions to understand the video's perspective and avoid misidentification.
Why this step is critical: Visual evidence alone can be ambiguous. For example, seeing streaming platform with a BRB screen could mean either:
- The user is streaming and stepped away (creator perspective)
- The user is watching someone else's stream (viewer perspective)
Only the user can clarify their relationship to the content.
Workflow: Analyze First, Then Ask
Step 1: Complete Initial Data Gathering
Before asking any questions, run the full analysis pipeline through Step 3c:
- ✓ Extract frames (Step 1)
- ✓ Extract and transcribe audio (Steps 2, 3a)
- ✓ Analyze audio with Gemini (Step 3b)
- ✓ Identify music with Shazam if detected (Step 3c)
Step 2: Build Contextual Evidence
Review what was gathered:
- Visual evidence: Sample 3-5 frames - what's visible? (UI, people, environments, screen content)
- Audio evidence: What's the audio content? (speech, music, ambient sounds)
- Music evidence: What songs were identified? With what timestamps?
- Conversation context: What has the user said about this video?
- User knowledge: What do you know about this user from conversation history and preferences?
Step 3: Ask Context Question (Q1) with Evidence
Present Q1 with context from your gathered evidence to help the user answer:
# Build context string from evidence
# Adapt summary based on what was found in the video
# Example 1: Stream Viewer
evidence_summary = """I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57
"""
# Example 2: Tutorial/How-to
evidence_summary = """I've analyzed the video and found:
- Visual: Person's hands visible working with materials, step-by-step demonstrations
- Audio: Instructional narration explaining process, background music
- Duration: 8:34
"""
# Example 3: Screen Recording (Software Demo)
evidence_summary = """I've analyzed the video and found:
- Visual: VS Code editor with code visible, cursor movements, terminal commands
- Audio: Voiceover explaining code changes, keyboard typing sounds
- Duration: 5:12
"""
# Example 4: Screen Recording (Gameplay)
evidence_summary = """I've analyzed the video and found:
- Visual: Game interface with HUD elements, character movement, gameplay
- Audio: Game sounds, background music, occasional commentary
- Duration: 12:45
"""
# Example 5: Personal/Casual
evidence_summary = """I've analyzed the video and found:
- Visual: Person on camera in home setting, handheld/phone camera movement
- Audio: Person speaking directly to camera, ambient room sounds
- Duration: 2:18
"""
# Example 6: Personal/Casual (Phone Clip)
evidence_summary = """I've analyzed the video and found:
- Visual: Quick clip of outdoor scene, vertical/portrait orientation, casual framing
- Audio: Ambient sounds, brief speech, wind noise
- Duration: 0:43
"""
# Example 7: Event/Performance
evidence_summary = """I've analyzed the video and found:
- Visual: Stage with performers, audience visible, concert venue setting
- Audio: Live music performance, crowd noise, applause
- Duration: 3:56
"""
# Example 8: Presentation/Slides
evidence_summary = """I've analyzed the video and found:
- Visual: Slide deck with bullet points and diagrams, presenter occasionally visible
- Audio: Presenter speaking, slide transition sounds
- Duration: 18:24
"""
# Example 9: Casual Documentation (Workspace)
evidence_summary = """I've analyzed the video and found:
- Visual: Desk setup with monitors and equipment, informal camera angles
- Audio: Background music, ambient office sounds, no narration
- Duration: 1:34
"""
# Example 10: Behind-the-Scenes
evidence_summary = """I've analyzed the video and found:
- Visual: Production equipment, camera setup, people working on set
- Audio: Technical discussions, equipment sounds, music playing
- Duration: 4:27
"""
# Template for implementation:
# Select the most relevant description based on gathered evidence
def build_evidence_summary(frames_analysis, audio_analysis, music_data, duration):
"""
Generate evidence summary based on analyzed content
Returns contextual summary that helps user identify video type
"""
visual_desc = describe_visual_content(frames_analysis)
audio_desc = describe_audio_content(audio_analysis, music_data)
return f"""I've analyzed the video and found:
- Visual: {visual_desc}
- Audio: {audio_desc}
- Duration: {format_duration(duration)}
"""
# Present Q1 with evidence context
AskUserQuestion(
questions=[
{
"question": f"{evidence_summary}\n\nWhat type of video is this?",
"header": "Video Type",
"multiSelect": false,
"options": [
{
"label": "Tutorial/How-to",
"description": "Teaching or demonstrating something step-by-step"
},
{
"label": "Personal",
"description": "Personal video, daily life content, a moment or activity, clip from phone"
},
{
"label": "Screen recording",
"description": "Recording your own screen (software demo, debugging video, gameplay, etc.)"
},
{
"label": "Watching something",
"description": "Recording yourself viewing someone else's content"
},
{
"label": "Event/Performance",
"description": "Concert, presentation, ceremony, or live event"
},
{
"label": "Other/Not sure",
"description": "Type a custom description of what this video shows"
}
]
}
]
)
Key Patterns in Evidence Summaries:
-
Be specific about what's visible: Not just "screen content" but "VS Code editor with code" or "Streaming platform interface with BRB screen"
-
Mention identifying audio: "Music playing (song identified)" vs. "instructional narration" vs. "live performance audio"
-
Note video characteristics: Portrait orientation, handheld movement, production quality
-
Keep it concise: 2-3 bullet points max, user can read it quickly
-
Help user self-identify: Evidence should make the correct option obvious
Step 4: Generate Smart Follow-up (Q2) Based on Evidence + Q1 Answer
CRITICAL: Generate Q2 dynamically based on:
- ✓ User's Q1 answer
- ✓ Visual evidence from frames
- ✓ Audio evidence from transcription/Gemini
- ✓ Music identification results
- ✓ Conversation context
- ✓ User preferences and history
Skip Q2 entirely if:
- The evidence makes the context obvious
- Q1 answer is self-explanatory (Tutorial, Event/Performance)
- You're highly confident about the analysis direction
Generate custom Q2 if needed:
If Q1 = "Watching something" AND you identified what they were watching:
# Example: You saw streaming platform + identified music
AskUserQuestion(
questions=[
{
"question": "I can see streaming platform with a BRB screen and identified 'Song Title' by Artist Name playing in the background. Is this what was happening?",
"header": "Confirm Context",
"multiSelect": false,
"options": [
{
"label": "Yes, that's correct",
"description": "Watching a stream with music playing in my space"
},
{
"label": "Partially correct",
"description": "Some of that is right, but let me clarify"
},
{
"label": "No, different context",
"description": "That's not quite what was happening"
}
]
}
]
)
If Q1 = "Watching something" AND you're uncertain what:
AskUserQuestion(
questions=[
{
"question": "What were you watching?",
"header": "Content Source",
"multiSelect": true, # Allow multiple selections
"options": [
{
"label": "Live stream",
"description": "Streaming platforms, Twitch, YouTube live, etc."
},
{
"label": "Video content",
"description": "YouTube video, movie, TV show, etc."
},
{
"label": "Music playing",
"description": "Music was playing in your space"
},
{
"label": "Other media",
"description": "Game, presentation, or other content"
}
]
}
]
)
If Q1 = "Personal" AND video shows workspace/music:
# You already know music was playing and workspace is visible
# Generate confirmation question instead of generic tags
AskUserQuestion(
questions=[
{
"question": "I can see your workspace and 'Song Title' playing. What tags describe this video?",
"header": "Context Tags",
"multiSelect": true,
"options": [
{
"label": "Music playing",
"description": "Background music in your space"
},
{
"label": "Workspace/setup",
"description": "Showing desk, equipment, or environment"
},
{
"label": "Activity in progress",
"description": "Doing something while recording"
},
{
"label": "Artistic/experimental",
"description": "Creative or artistic intent"
}
]
}
]
)
If Q1 = "Screen recording":
# Only ask if genuinely uncertain from frames
# You likely already saw what's on screen
AskUserQuestion(
questions=[
{
"question": "I can see {specific_app_or_content}. What were you recording?",
"header": "Screen Content",
"multiSelect": false,
"options": [
{
"label": "Software/application",
"description": "Demonstrating a program or workflow"
},
{
"label": "Gameplay",
"description": "Playing a game"
},
{
"label": "Design/creative work",
"description": "Working in design tools, editors, etc."
},
{
"label": "General computer use",
"description": "Browsing, chatting, or mixed activities"
}
]
}
]
)
If Q1 = "Tutorial/How-to" or "Event/Performance":
→ Skip Q2 - these are self-explanatory, proceed to analysis
If Q1 = "Other/Not sure" (custom text provided):
→ Use the custom text to inform analysis, skip Q2 unless truly necessary
Store and Use Metadata
After Q1 (and Q2 if asked), store structured metadata:
video_metadata = {
"video_type": user_q1_answer, # "Watching something"
"perspective": derived_perspective, # "First-Person Viewer"
"content_tags": user_q2_answers, # ["live stream", "music playing"]
"evidence": {
"visual": visual_summary, # From frames
"audio": audio_summary, # From Gemini
"music": music_identification, # From Shazam
"duration": video_duration
}
}
Use this metadata throughout remaining analysis:
- Guide visual analysis focus (Step 5)
- Inform subagent prompts
- Set appropriate terminology in final summary
- Avoid misidentifying perspective or intent
Example: Context-Aware Question Flow
Scenario: User shares video showing streaming platform with BRB screen, person visible, music playing
Data Gathered (Steps 1-3d):
- Frames: Streaming platform UI, BRB overlay, person, workspace visible
- Audio: Music "Song Title" by Artist Name (0:32-1:57), minimal speech
- Duration: 1:57
- Context: User information from conversation history
Q1 with Context:
I've analyzed the video and found:
- Visual: Streaming platform interface with BRB screen, person visible, workspace shown
- Audio: Music playing ("Song Title" by Artist Name), minimal speech
- Duration: 1:57
What type of video is this?
→ User selects: "Watching something"
Smart Q2 Generation:
# You already know:
# - Streaming platform with BRB = watching a stream
# - Music identified = "Song Title" playing in their space
# - Person visible = user on camera
# Instead of generic "What were you watching?", generate specific confirmation:
"I can see streaming platform with a BRB screen and identified 'Song Title' by Artist Name
playing in the background. Were you watching someone's stream while music played in your space?"
Options:
- Yes, that's correct
- Partially - let me clarify
- No, different context
Result: Only 2 questions needed, and Q2 is informed and specific rather than generic.
Alternative - High Confidence, Skip Q2:
If extremely confident from evidence, skip Q2 entirely:
Q1 answer: "Watching something"
→ Skip Q2, proceed with metadata:
{
video_type: "Watching something",
perspective: "First-Person Viewer",
content_tags: ["live stream", "music playing", "workspace visible"]
}
Benefits of This Approach
- Evidence-informed questions: Q1 includes what you found, helping user answer
- Smart Q2 generation: Tailored to what you already know + user's Q1 answer
- Skip when confident: Don't ask unnecessary questions if evidence is clear
- Confirmation over discovery: Ask "Is this correct?" when you have strong evidence
- Context-aware: Uses conversation history, user preferences, and gathered data
- Prevents misidentification: Clarifies creator vs. viewer perspective early
- Efficient: Maximum 2 questions, often just 1
Integration with Remaining Workflow
After completing Step 3.5 (Context Clarification), proceed to:
Step 4: Video Type Detection and Analysis Strategy
- Use
video_metadata.perspective to select appropriate analysis focus
- Apply type-specific guidance based on clarified context
Step 5: Visual Analysis with video-frame-analyzer Subagent(s)
- Pass
video_metadata to subagent prompts
- Ensure analysis matches user's stated relationship to content
- Use correct terminology (viewer/creator/observer language)
The metadata from context clarification informs all downstream analysis.
4. Video Perspective and Type Detection (UPDATED)
After context clarification (Step 3.5), use the user's answers and gathered evidence to determine the appropriate analysis approach.
Two-Dimensional Classification System
Dimension 1: Perspective (Who is filming and why?)
Dimension 2: Content Type (What's in the video?)
Perspective Classification
Derive perspective from user's Q1 answer and evidence:
| User's Q1 Answer | Perspective | Analysis Approach |
|---|
| Tutorial/How-to | First-Person Creator | User is teaching/presenting |
| Personal | First-Person Creator OR Meta/Hybrid | User is documenting their own life/activities |
| Screen recording | First-Person Creator | User is demonstrating their own screen |
| Watching something | First-Person Viewer | User is consuming someone else's content |
| Event/Performance | Third-Person Observer | User is filming someone/something else |
| Other/custom text | Derive from description | Analyze custom text to determine |
Expanded Content Type Taxonomy
Based on perspective, select the appropriate content type analysis:
First-Person Viewer Types
The user is watching/consuming content created by others
Stream Viewer
Indicators:
- Visual: Streaming platform UI (Discord, Twitch, YouTube), others' stream content, BRB screens, chat visible
- Audio: Music in viewer's space, ambient sounds, minimal commentary
- Evidence: Streaming platform visible, user not the presenter/streamer
Analysis Focus:
- Primary: What stream/content is being watched? (streamer name, stream title, platform)
- Track: BRB screens, waiting periods, stream states, chat activity
- Note: Viewer's environment (workspace, room, objects visible)
- Look For:
- Parallel activities (what is viewer doing while watching?)
- Music or ambient audio in viewer's space (distinct from stream audio)
- Environmental context and mood
- Correlation between stream content and viewer behavior
Reaction Video
Indicators:
- Visual: Split screen (content + reactor) or switches between content and reactions
- Audio: Commentary over content, reactions, pauses for analysis
- Evidence: User visible reacting to visible content
Analysis Focus:
- Primary: What content is being reacted to? User's reactions and commentary
- Track: Reaction timing, pauses, emotional responses
- Note: User's analysis, critiques, or engagement with content
- Look For: Key moments that trigger reactions, commentary themes
Viewing Documentation
Indicators:
- Visual: Browser/player UI, content consumption in progress, informal framing
- Audio: Background music, ambient sounds, casual environment
- Evidence: Passive viewing being documented
Analysis Focus:
- Primary: What's being consumed, why is it being documented
- Track: Viewing progression, pauses, environmental changes
- Note: Mood, atmosphere, context for documentation
- Look For: Artistic intent, thematic connections
First-Person Creator Types
The user is creating/presenting content
Tutorial/How-to
Indicators:
- Visual: Step-by-step demonstrations, hands visible, process shown
- Audio: Instructional narration, explanations, process sounds
- Evidence: Deliberate teaching structure
Analysis Focus:
- Primary: What's being taught, step-by-step process
- Track: Instructions, materials/tools, technique demonstrations
- Note: Key concepts, warnings, tips
- Look For: Learning objectives, common mistakes addressed
Screen Recording (Creator)
Indicators:
- Visual: User's own screen with deliberate navigation, cursor movements, UI interactions
- Audio: Voiceover tutorial, keyboard clicks, narration
- Evidence: User demonstrating their own workflow
Analysis Focus:
- Primary: UI elements, workflow steps, applications used
- Track: Mouse/cursor position, clicked elements, opened applications
- Note: Workflow steps, error messages, command sequences
- Look For: Transitions between applications, file operations, settings changes
Personal (Vlog/Life Content)
Indicators:
- Visual: Person on camera, home/outdoor settings, handheld or mounted camera
- Audio: Direct-to-camera speech, personal narrative, ambient sounds
- Evidence: User sharing personal moments or experiences
Analysis Focus:
- Primary: Setting/location, what person is saying/doing
- Track: Emotional tone, activities, locations visited
- Note: Personal details, narrative arc, objects/items shown
- Look For: Story progression, mood shifts, environment changes
Presentation/Slides
Indicators:
- Visual: Slide layouts, bullet points, diagrams, presenter view
- Audio: Presenter speaking, slide advance sounds
- Evidence: Formal presentation structure
Analysis Focus:
- Primary: Slide content (headings, bullet points, data)
- Track: Slide numbers, transitions, presenter appearance
- Note: Key concepts, visualizations, quotes, formulas
- Look For: Slide themes, speaker gestures, main arguments
Meta/Hybrid Types
The user is documenting the process of creating/consuming
Casual Documentation
Indicators:
- Visual: Informal framing, switches between activities, workspace visible, no clear structure
- Audio: Ambient audio, music, casual speech, environmental sounds
- Evidence: Informal recording of moments/activities
Analysis Focus:
- Primary: Activities happening, personal environment, objects/items
- Track: Environmental changes, mood shifts, items introduced
- Note: Personal details that provide context, atmosphere
- Look For:
- Documentary-style observations
- Authentic, unscripted moments
- Connections between visual, audio, and thematic elements
- Artistic or experimental intent
Behind-the-Scenes
Indicators:
- Visual: Production equipment visible, setup shots, workspace, process in action
- Audio: Technical discussions, process sounds, tools being used
- Evidence: Documenting creation workflow
Analysis Focus:
- Primary: Creation process, tools/equipment, methodology
- Track: Setup changes, workflow steps, technical decisions
- Note: Challenges encountered, solutions implemented
- Look For: Insights into creative/production process
Process Video
Indicators:
- Visual: Work-in-progress visible, tools/software, iterative changes
- Audio: Narration of process, thinking out loud, work sounds
- Evidence: Documenting work as it happens
Analysis Focus:
- Primary: Steps taken, decision-making, progress
- Track: Iterations, changes, refinements
- Note: Reasoning, challenges, solutions
- Look For: Learning moments, key decisions
Third-Person Observer Types
The user is filming someone/something else
Event/Performance
Indicators:
- Visual: Stage/venue, audience, performers, event setting
- Audio: Live audio, applause, ambient event noise
- Evidence: User filming an event they're attending
Analysis Focus:
- Primary: Performers, performance content, venue
- Track: Key performance moments, crowd reactions, technical elements
- Note: Event highlights, special effects, atmosphere
- Look For: Memorable moments, mistakes, energy shifts
Interview/Podcast
Indicators:
- Visual: Two+ people, static camera, simple background
- Audio: Conversational dialogue, minimal music
- Evidence: Discussion/interview being filmed
Analysis Focus:
5. Visual Analysis with video-frame-analyzer Subagent(s)
After frames are extracted AND video type is assessed, delegate to the specialized video-frame-analyzer subagent(s) for systematic visual analysis:
Single Subagent (< 30 frames):
For videos with fewer than 30 frames, use a single subagent:
IMPORTANT: Use the full qualified subagent name: video-toolkit:video-frame-analyzer
Step 1: Assess Video Type (sample first 3-5 frames)
Before full analysis, quickly review a few early frames to determine video type and appropriate analysis focus:
Use the video-toolkit:video-frame-analyzer subagent to assess video type from /tmp/video-toolkit-[timestamp]/frames
Sample frames 1-5 and determine:
1. Video type (screen recording, vlog, presentation, music video, tutorial, etc.)
2. Recommended analysis focus based on type (see Type-Specific Analysis Focus section)
3. Key elements to track throughout the video
Save assessment to: /tmp/video-toolkit-[timestamp]/video-type-assessment.md
Step 2: Full Frame Analysis (with type-adaptive prompts)
Use the video-toolkit:video-frame-analyzer subagent to analyze the frames in /tmp/video-toolkit-[timestamp]/frames
**Video Type Detected:** [Insert detected type from assessment]
The following files are available:
- Frames: /tmp/video-toolkit-[timestamp]/frames/*.png (X frames total)
- Metadata: /tmp/video-toolkit-[timestamp]/frames/frames_metadata.json (contains frame timestamps)
- Transcript: /tmp/video-toolkit-[timestamp]/transcript.md (if video has audio)
- Audio Analysis: /tmp/video-toolkit-[timestamp]/audio_analysis.md (Gemini audio understanding, if available)
- Music Identification: /tmp/video-toolkit-[timestamp]/music_identification.md (Shazam results, if music detected)
- Video Type Assessment: /tmp/video-toolkit-[timestamp]/video-type-assessment.md
Based on the detected video type, apply appropriate analysis focus:
[Insert type-specific focus points from Type-Specific Analysis Focus section]
Analyze all frames, correlate visual content with transcript/audio using timestamps, and pay special attention to elements relevant for this video type.
Save your complete analysis to: /tmp/video-toolkit-[timestamp]/video-analysis.md
Parallel Subagents (≥ 30 frames):
For videos with 30+ frames, spawn multiple video-toolkit:video-frame-analyzer subagents in parallel, each handling a subset of frames:
IMPORTANT:
- Use the full qualified subagent name:
video-toolkit:video-frame-analyzer
- Use a single message with multiple Task tool calls to run agents in parallel
Example for 50 frames:
# Spawn 5 subagents in parallel, each analyzing 10 frames
# Agent 1: frames 1-10
# Agent 2: frames 11-20
# Agent 3: frames 21-30
# Agent 4: frames 31-40
# Agent 5: frames 41-50
How to split frames:
- 30-50 frames: 3-5 subagents (10 frames each) - launch all in parallel
- 50-100 frames: 5-10 subagents (10 frames each) - launch in waves of 5
- 100+ frames: Consider using interval mode instead, or max 10 subagents in waves
Staged Execution for Long Videos:
For videos requiring more than 5 subagents, launch them in waves to avoid overwhelming the system:
Example: 100 frames = 10 subagents
- Wave 1: Launch subagents 1-5 in parallel (frames 1-50)
- Wait for Wave 1 to complete
- Wave 2: Launch subagents 6-10 in parallel (frames 51-100)
- Wait for Wave 2 to complete
- Synthesize all results
Why staged execution?
- Prevents too many concurrent subagents
- Better resource management
- Easier to debug if something fails
- More predictable performance
Prompt for parallel subagents:
IMPORTANT:
- First, assess video type with a quick subagent analyzing frames 1-5
- Then create the
partial-analyses/ directory: mkdir -p /tmp/video-toolkit-[timestamp]/partial-analyses
- Launch parallel subagents with type-aware prompts
Step 1: Quick Video Type Assessment
Use the video-toolkit:video-frame-analyzer subagent to assess video type from frames 1-5 in /tmp/video-toolkit-[timestamp]/frames
Sample the first 5 frames and determine:
1. Video type (screen recording, vlog, presentation, music video, tutorial, interview, documentary, gameplay, event, animation)
2. Recommended analysis focus based on type
3. Key elements to track throughout
Review audio analysis if available to help determine type.
Save assessment to: /tmp/video-toolkit-[timestamp]/video-type-assessment.md
Step 2: Type-Adaptive Parallel Analysis
After video type is assessed, use this prompt template for each parallel subagent:
Use the video-toolkit:video-frame-analyzer subagent to analyze frames [start]-[end] from /tmp/video-toolkit-[timestamp]/frames
**Video Type:** [Insert type from assessment]
**Analysis Focus:** [Insert relevant focus points for this type]
Files available:
- Frames: frame_00[start].png through frame_00[end].png
- Metadata: /tmp/video-toolkit-[timestamp]/frames/frames_metadata.json
- Transcript: /tmp/video-toolkit-[timestamp]/transcript.md (if available)
- Audio Analysis: /tmp/video-toolkit-[timestamp]/audio_analysis.md (if available)
- Video Type Assessment: /tmp/video-toolkit-[timestamp]/video-type-assessment.md
Focus on frames [start] through [end] only. Based on video type, provide:
1. Timestamp range covered
2. Visual content description (emphasize type-specific elements)
3. Key elements for this video type (UI/text for screen recordings, actions/setting for vlogs, etc.)
4. Scene changes within your range
5. Correlation with transcript/audio (if applicable)
Save your analysis to: /tmp/video-toolkit-[timestamp]/partial-analyses/part[N].md
Example for Vlog/Personal Video:
Use the video-toolkit:video-frame-analyzer subagent to analyze frames 1-12 from /tmp/video-toolkit-[timestamp]/frames
**Video Type:** Vlog/Personal
**Analysis Focus:**
- Primary: Setting/location, people present, activities happening
- Track: Emotional tone, interactions, objects/items shown
- Note: What person is doing, wearing, holding
- Look For: Personal details (smoking, eating, gestures), environment changes
Files available:
- Frames: frame_0001.png through frame_0012.png
- Metadata: /tmp/video-toolkit-[timestamp]/frames/frames_metadata.json
- Transcript: /tmp/video-toolkit-[timestamp]/transcript.md
- Audio Analysis: /tmp/video-toolkit-[timestamp]/audio_analysis.md
- Video Type Assessment: /tmp/video-toolkit-[timestamp]/video-type-assessment.md
Focus on frames 1-12. Provide:
1. Timestamp range: 0:00 - 0:22
2. Visual content: Describe setting, people, activities with attention to personal details
3. Personal elements: Note smoking, gestures, objects held, clothing, emotional state
4. Scene/environment changes within this range
5. Correlation with what's being said (transcript) and music/ambient audio
Save your analysis to: /tmp/video-toolkit-[timestamp]/partial-analyses/part1.md
Example Task tool invocation:
# First create the directory
Bash("mkdir -p /tmp/video-toolkit-[timestamp]/partial-analyses")
# Then spawn parallel subagents
Task(
subagent_type="video-toolkit:video-frame-analyzer",
description="Analyze frames 1-13",
prompt="Analyze frames 1-13... Save to: /tmp/video-toolkit-[timestamp]/partial-analyses/part1.md"
)
After parallel analysis completes:
- Read all partial analyses from
partial-analyses/ directory (part1.md, part2.md, etc.)
- Synthesize into a unified comprehensive summary file at
/tmp/video-toolkit-[timestamp]/video-analysis.md
- Correlate findings across all sections
- Clean up:
rm -rf /tmp/video-toolkit-[timestamp]/partial-analyses
- Save comprehensive analysis alongside source video (see step 7 in Combined Summary Generation below)
- Present final summary to user
What the subagent(s) do:
-
Read frames_metadata.json for accurate frame-to-timestamp mapping
-
Read transcript.md (if available) for audio content
-
Systematically analyze assigned frames using vision capabilities
-
Correlate visual content with spoken dialogue using timestamps
-
Identify video type, UI elements, text, and visual patterns
-
Generate analysis with multimodal correlation
-
Note: Important quotes, key points, visual aids
-
Look For: Emotional moments, agreements/disagreements
Documentary
Indicators:
- Visual: B-roll footage, archival material, location shots, graphics
- Audio: Narrator voiceover, ambient audio, music
- Evidence: Documentary storytelling structure
Analysis Focus:
- Primary: B-roll content, archival material, locations
- Track: Text overlays (names, dates, locations), graphics
- Note: Historical context, expert interviews, data
- Look For: Primary sources, reenactments, maps/timelines
Using the Classification
After determining perspective and content type from Step 3.5 context clarification:
- Select the appropriate analysis focus from the tables above
- Apply type-specific guidance to visual analysis (Step 5)
- Use correct terminology in subagent prompts and final summary
- Avoid misidentification by respecting the user's stated relationship to the content
Example for "Stream Viewer" (First-Person Viewer):
video_metadata = {
"video_type": "Watching something",
"perspective": "First-Person Viewer",
"content_type": "Stream Viewer",
"context_tags": ["live stream", "music playing"],
"analysis_focus": {
"primary": "Stream content being watched, viewer's environment",
"track": "BRB screens, stream states, workspace, parallel activities",
"note": "Music in viewer's space, environmental context, mood",
"look_for": "Correlation between stream and viewer behavior, thematic connections"
}
}
This metadata guides all subsequent analysis steps.
Restructured Workflow: Optional Song Theme Research
Current Issue
Song theme research (Step 3d) currently happens automatically after music identification, but:
- Not all videos need thematic music analysis
- Tutorial with background music? Music is just ambiance
- Screen recording with Spotify playing? Not relevant to content
- Wastes time/tokens when music isn't thematically important
Solution: Move to After Context Clarification
New sequence:
- Frame Extraction
- Audio Extraction
- Audio Analysis (Sequential Workflow)
- 3a. Speech Transcription (Whisper)
- 3b. Audio Understanding (Gemini)
- 3c. Music Identification (Shazam - if music detected)
3d. Song Theme Research ← REMOVE FROM HERE
- Context Clarification (NEW - Step 3.5)
- Video Perspective and Type Detection
- OPTIONAL: Song Theme Research ← MOVE TO HERE (becomes Step 5.5)
- Visual Analysis with subagents
When to Research Song Themes (Conditional Logic)
Research song themes IF:
- Video type suggests thematic music importance
- User context indicates artistic/creative intent
- Music plays during significant portions of video
- Multiple perspective/content switches suggest deliberate choices
Skip song theme research IF:
- Video type is purely functional (tutorial, screen recording demo)
- Music is brief background ambiance
- User context suggests music is incidental
- Analysis focus doesn't benefit from song meaning
Step 5.5: Optional Song Theme Research (NEW)
When to Use: After determining video type and perspective (Steps 3.5 and 4), conditionally research song themes if music appears thematically relevant.
Decision Logic
Research song themes for these video types:
| Video Type | Perspective | When to Research | Reasoning |
|---|
| Stream Viewer | First-Person Viewer | ✅ USUALLY | Music in viewer's space may have thematic connection to stream content or mood |
| Casual Documentation | Meta/Hybrid | ✅ USUALLY | Artistic/creative videos often use music deliberately |
| Personal | First-Person Creator | ✅ IF ARTISTIC | Only if video seems artistic/experimental vs. casual life clip |
| Music Video/Creative | First-Person Creator | ✅ ALWAYS | Music is the primary content |
| Vlog/Personal | First-Person Creator | ⚠️ CONDITIONAL | Only if music plays significant role |
| Behind-the-Scenes | Meta/Hybrid | ⚠️ CONDITIONAL | Only if music seems intentionally chosen |
| Tutorial/How-to | First-Person Creator | ❌ SKIP | Music is background, not thematic |
| Screen Recording | First-Person Creator | ❌ SKIP | Music is incidental (Spotify/background) |
| Presentation | First-Person Creator | ❌ SKIP | Music not relevant to content |
| Event/Performance | Third-Person Observer | ❌ SKIP | Music is part of event, not selected |
Conditional Check
Before researching song themes, evaluate:
def should_research_song_themes(video_metadata, music_data):
"""
Determine if song theme research would benefit analysis
Args:
video_metadata: From Step 3.5 context clarification
music_data: From Step 3c music identification
Returns:
bool: True if research would be valuable
"""
# No music detected? Skip
if not music_data.get("has_music"):
return False
# Check video type thematic relevance
video_type = video_metadata["video_type"]
perspective = video_metadata["perspective"]
# Always research for these types
always_research = [
"Music Video/Creative",
"Watching something" + "First-Person Viewer", # Stream viewer
"Casual documentation" + "Meta/Hybrid"
]
# Never research for these types
skip_research = [
"Tutorial/How-to",
"Screen recording" + "First-Person Creator",
"Presentation",
"Event/Performance"
]
# Check if video type + perspective matches "always" patterns
if any(pattern in f"{video_type} + {perspective}" for pattern in always_research):
return True
# Check if video type matches "skip" patterns
if any(pattern in video_type for pattern in skip_research):
return False
# Conditional cases - check additional factors
# Music duration significant? (>30% of video)
music_duration = sum(segment["duration"] for segment in music_data["music_segments"])
video_duration = video_metadata["evidence"]["duration"]
if music_duration / video_duration > 0.3:
return True # Music plays significant role
# User context suggests artistic intent?
context_tags = video_metadata.get("context_tags", [])
if "artistic/experimental" in context_tags:
return True
# Default: skip unless evidence suggests thematic importance
return False
Implementation
If research is warranted:
if should_research_song_themes(video_metadata, music_data):
# Research themes for each identified song
for song in music_data["songs"]:
query = f"{song['title']} {song['artist']} lyrics meaning themes"
# Try Perplexity first (better for analysis)
try:
results = mcp__perplexity_search_web(query, recency="year")
song["themes"] = extract_themes(results)
except:
# Fallback to WebSearch if Perplexity unavailable
results = WebSearch(query)
song["themes"] = extract_themes(results)
# Store theme research in metadata
video_metadata["song_themes"] = {
song["title"]: song["themes"] for song in music_data["songs"]
}
else:
# Skip research, note music was identified but themes not needed
video_metadata["song_themes"] = None
# Music identification results still available in music_data
Communicate decision to user:
✓ Music identified: "Song Title" by Artist Name (2009)
✓ Researching song themes (video appears to use music thematically)
→ Themes: song themes and meaning
OR
✓ Music identified: "Background Music Track" (ambient)
⊘ Skipping theme research (background music, not thematically relevant)
Benefits
- Efficiency: Don't waste time/tokens on irrelevant research
- Precision: Only research when it enhances understanding
- User experience: Clear communication about what's happening
- Flexibility: Easy to adjust criteria based on video type
Example Decisions
Example 1: Stream Viewer Video (Em's case)
- Video type: "Watching something" (Stream Viewer)
- Music: "Song Title" playing 0:32-1:57 (85% of video duration)
- Decision: ✅ Research themes - music significant + viewer perspective
- Result: Themes inform understanding of why this song was chosen
Example 2: Coding Tutorial
- Video type: "Tutorial/How-to"
- Music: Lo-fi beats playing throughout
- Decision: ❌ Skip research - background ambiance for focus
- Result: Music noted but themes not analyzed
Example 3: Personal Phone Clip
- Video type: "Personal"
- Music: 10 seconds of radio in background
- Context tags: None suggesting artistic intent
- Music duration: 10s / 120s = 8%
- Decision: ❌ Skip research - incidental background music
- Result: Music noted but not analyzed
Example 4: Artistic Video Essay
- Video type: "Personal"
- Music: Specific song playing throughout
- Context tags: ["artistic/experimental"]
- Music duration: 90s / 100s = 90%
- Decision: ✅ Research themes - artistic intent + significant duration
- Result: Themes analyzed for artistic interpretation
Updated SKILL.md Structure
**3. Audio Analysis (Sequential Workflow)**
- 3a. Speech Transcription (Whisper)
- 3b. Audio Understanding (Gemini)
- 3c. Music Identification (Shazam - if music detected)
*6. Combined Summary Generation**
After the video-frame-analyzer completes its analysis, synthesize all multimodal insights:
**Available Analysis Files:**
- `video-type-assessment.md` - Video type and analysis strategy
- `transcript.md` - Speech transcription (Whisper)
- `audio_analysis.md` - Audio understanding (Gemini)
- `music_identification.md` - Identified songs (Shazam, if music detected)
- `video-analysis.md` - Visual analysis (video-frame-analyzer, type-aware)
- `reconciled_transcript.md` - Reconciled transcript (optional, created by comparing Whisper + Gemini)
**Synthesis Steps:**
1. Review video type assessment to understand analysis context
2. Review visual analysis from video-frame-analyzer (type-aware analysis)
3. If music was identified, review song themes from web search for thematic context
4. **Compare and reconcile transcripts** (OPTIONAL):
- **ALWAYS spawn a Task subagent (general-purpose) to compare transcripts** (reduces token usage in main thread)
- The subagent should:
- Read both `transcript.md` (Whisper) and `audio_analysis.md` (Gemini) for speech content
- Note that Whisper specializes in speech transcription but can mishear words
- Note that Gemini Audio API also transcribes speech and may catch errors Whisper missed
- **Compare the two transcripts** and identify any discrepancies
- Use context, grammar, and semantic meaning to determine which is more accurate
- Create a reconciled transcript that uses the most accurate version
- Document any significant differences and reasoning for choices
- Save the reconciled transcript to `/tmp/video-toolkit-[timestamp]/reconciled_transcript.md`
- After subagent completes, read the reconciled transcript for use in synthesis
3. Review audio analysis for non-speech sounds and music description
4. If music was identified, review song details with timestamps
5. **Correlate all insights using timestamps** to create unified narrative
6. Identify synchronization between:
- What's visible (frames)
- What's said (Whisper)
- What's heard (Gemini audio)
- What music is playing (Shazam)
7. **IMPORTANT: Save comprehensive analysis alongside source video**
- Copy `/tmp/video-toolkit-[timestamp]/video-analysis.md` to replace the placeholder analysis file
- Target location: `<video-directory>/<video-name>-analysis.md`
- This ensures the detailed analysis persists alongside the source video
- Example: `cp /tmp/video-toolkit-1234567890/video-analysis.md "/path/to/video-analysis.md"`
8. Present comprehensive multimodal summary to user
*7. Discussion and Follow-up**
After generating summary:
- Present findings in markdown format
- Enable conversational follow-up questions
- Offer to extract specific clips or moments
- Provide additional analysis on request
### Video Editing Workflow
For video editing operations, use `scripts/edit_video.py`:
**Clip Video (Extract Segment)**
```bash
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/edit_video.py clip <input_video> <start_time> <end_time> <output_video>
start_time: Format HH:MM:SS or MM:SS or secondsend_time: Format HH:MM:SS or MM:SS or seconds- Example:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/edit_video.py clip video.mp4 00:01:30 00:02:45 clip.mp4
Merge Videos (Concatenate)
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/edit_video.py merge <output_video> <input1> <input2> [input3...]
- Concatenates multiple videos in order
- Videos should have same codec/resolution for best results
- Example:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/edit_video.py merge final.mp4 part1.mp4 part2.mp4 part3.mp4
Split Video (By Duration)
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/edit_video.py split <input_video> <segment_duration> <output_prefix>
segment_duration: Length of each segment in secondsoutput_prefix: Prefix for output files (e.g., "segment" → segment_001.mp4, segment_002.mp4)- Example:
${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/.venv/bin/python3 ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/edit_video.py split long_video.mp4 300 chunk
Orchestration Script (All-in-One Analysis)
For comprehensive analysis, use the orchestration wrapper with clear flag-based arguments:
# Basic usage (uses interval mode with 2s default - safest for most videos)
bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_video.sh video.mp4
# Interval mode with custom interval
bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_video.sh video.mp4 --interval 3
# Scene detection mode (for videos with clear scene changes like movies)
bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_video.sh video.mp4 --mode scene-detect --threshold 0.05
# With custom whisper model
bash ${CLAUDE_PLUGIN_ROOT}/skills/video-toolkit/scripts/analyze_video.sh video.mp4 --interval 2 --whisper-model small
Available Options:
--mode <scene-detect|interval> - Frame extraction mode (default: interval)--threshold <value> - Scene detection threshold 0.0-1.0 (default: 0.02)--interval <seconds> - Interval between frames (default: 2)--whisper-model <model> - Whisper model: tiny.en, base, small, medium, large (default: base)--help - Show usage information
This script:
- Creates temporary working directory
- Extracts frames at specified interval
- Extracts audio
- Transcribes speech with Whisper (local)
- Analyzes audio with Gemini (API)
- If music detected, identifies songs with Shazam (API)
- Organizes results in structured format
- Returns paths to all artifacts
Returns:
frames/: Directory of extracted framesframes/frames_metadata.json: Frame timestampsaudio.wav: Extracted audio filetranscript.md: Speech transcription (Whisper)audio_analysis.md: Audio understanding (Gemini)music_identification.md: Song identification (Shazam, if music detected)analysis-summary.md: Overview and next steps
Bundled Resources
Scripts
scripts/install_dependencies.sh
- Install FFmpeg, Python venv, Whisper, Gemini API, and Shazam
- One-time setup, run on first use
- Checks for existing installations
scripts/setup_api_keys.py
- Configure Gemini API key for audio analysis
- Configure Shazam API key (optional, uses public endpoint by default)
- Validates and saves API keys to config file
scripts/extract_frames.py
- FFmpeg-based frame extraction
- Configurable interval and scene detection modes
- Generates frames_metadata.json with timestamps
- Efficient processing for long videos
scripts/extract_audio.py
- Extract audio track to WAV format
- Handles various input codecs
- Whisper-compatible output
scripts/transcribe_audio.py
- Whisper integration for speech transcription
- Model selection support (tiny.en, base, small, medium, large)
- Timestamped output with metadata
- Runs locally (no API key required)
scripts/analyze_audio_gemini.py
- Gemini Audio API integration for comprehensive audio analysis
- Detects music with precise timestamps
- Identifies non-speech sounds and audio events
- Returns markdown + JSON with music segments
scripts/identify_music.py
- Shazam integration for music identification
- Extracts music segments based on Gemini timestamps
- Identifies songs with title, artist, album, genre
- Only runs if Gemini detected music
scripts/edit_video.py
- Unified editing tool (clip, merge, split)
- FFmpeg-based for reliability
- Error handling and validation
scripts/analyze_video.sh
- Orchestration wrapper for full analysis
- Persists final summary alongside source video
- Automatic cleanup of temporary files
- Structured output format
References
references/ffmpeg_commands.md
- Common FFmpeg commands and patterns
- Codec information and best practices
- Troubleshooting guide
File Management
Final Analysis Summary:
The complete analysis summary is automatically saved alongside the source video:
- Location: Same directory as the source video
- Filename:
<video-name>-analysis.md (e.g., test-analysis.md for test.mov)
- Format: Markdown with YAML frontmatter
- Persistence: Permanent (lives next to your video file)
Temporary Working Files:
During analysis, temporary files are created in /tmp/video-toolkit-{timestamp}/:
- Frames:
frames/ subdirectory with extracted PNG files
- Audio:
audio.wav (if video has audio)
- Transcript:
transcript.md with timestamped speech
- Audio Analysis:
audio_analysis.md (Gemini output)
- Music Identification:
music_identification.md (Shazam output, if music detected)
Cleanup Workflow:
After completing video analysis, ALWAYS use the AskUserQuestion tool to ask the user:
Question: "Would you like me to keep these analysis files, or should I clean them up?"
Options:
- "Clean up temporary files" → Run: rm -rf /tmp/video-toolkit-{timestamp}
(Keeps only the persisted summary: <video-name>-analysis.md)
- "Keep everything" → Inform user of /tmp location for frames/audio/transcripts
(User responsible for manual cleanup later)
Important Notes:
- The final summary (
<video-name>-analysis.md) is always persisted alongside the source video
- Temporary files in
/tmp contain frames, audio, transcripts (can be large)
- User may want to keep frames for visual analysis or audio for further processing
- If keeping files, remind user that
/tmp persists until reboot or ~3 days on macOS
Manual Cleanup Commands (rarely needed):
# Check if any temp files remain (shouldn't happen)
ls -lh /tmp/video-toolkit-*
# Clean all video-toolkit temp files (if automatic cleanup failed)
rm -rf /tmp/video-toolkit-*
# Check disk usage in /tmp
du -sh /tmp/video-toolkit-* 2>/dev/null || echo "No temp files found (good!)"
Output Format
Analysis Summary Template:
# Video Analysis: {filename}
## Overview
- **Duration**: MM:SS
- **Resolution**: WxH
- **File Size**: XX MB
- **Analysis Date**: YYYY-MM-DD
## Transcript
[Timestamped transcript from Whisper]
## Visual Analysis
### Scene 1 (00:00 - 00:XX)
- Frame descriptions
- Key moments
- Visual details
### Scene 2 (00:XX - 00:XX)
...
## Key Highlights
- Important moments
- Notable quotes
- Visual highlights
## Summary
Overall narrative and insights
---
*Generated by video-toolkit v1.0.0*
Examples
Example 1: Analyze Meeting Recording
User: Can you analyze this meeting recording and summarize the key points?
Claude:
1. Run analyze_video.sh on the meeting file
2. Extract frames every 5 seconds
3. Transcribe audio with Whisper (base model)
4. Analyze slides/screen sharing in frames
5. Identify speakers and topics from transcript
6. Generate summary with:
- Meeting agenda items
- Decisions made
- Action items
- Key discussion points
Example 2: Extract Highlight Clip
User: Extract the segment from 2:30 to 4:15 from this video
Claude:
python scripts/edit_video.py clip input.mp4 00:02:30 00:04:15 highlight.mp4
Example 3: Combine Multiple Clips
User: Merge these three video files into one
Claude:
python scripts/edit_video.py merge final_video.mp4 clip1.mp4 clip2.mp4 clip3.mp4
Troubleshooting
FFmpeg Not Found:
- Run
scripts/install_dependencies.sh
- Or install manually:
brew install ffmpeg (macOS) or apt install ffmpeg (Linux)
Whisper Model Download Slow:
- Models download on first use
- Models cached in
~/.cache/whisper/
Best Practices
-
Frame Interval Selection
- Action videos: 1-2 seconds
- Dialogue/presentations: 3-5 seconds
- Surveillance/static: 10+ seconds
-
Whisper Model Selection
- Quick analysis: tiny.en or base
- Important transcription: medium or large
- Non-English: Use multilingual models (not .en)
-
Temp File Management
- Clean up after analysis unless user asks to keep
- Monitor disk space for long videos
- Use
/tmp for automatic cleanup on reboot
-
Video Editing
- Keep original files safe (don't overwrite)
- Use descriptive output filenames
- Check codec compatibility for merging
Source: emdashcodes/claude-code-plugins — distributed by TomeVault.
