| name | analyze-video |
| description | Adds visual descriptions to transcripts by extracting and analyzing video frames with ffmpeg. Creates visual transcript with periodic visual descriptions of the video clip. Use when all files have audio transcripts present (transcript) but don't yet have visual transcripts created (visual_transcript). |
Skill: Analyze Video
Add visual descriptions to audio transcripts by extracting JPG frames with ffmpeg and analyzing them. Never read video files directly - extract frames first.
Prerequisites
Videos must have audio transcripts. Run transcribe-audio skill first if needed.
Workflow
1. Copy & Clean Audio Transcript
Don't read the audio transcript, just copy it and then prepare it by using the prepare_visual_script.rb file. This removes word-level timing data and prettifies the JSON for easier editing:
cp libraries/[library]/transcripts/video.json libraries/[library]/transcripts/visual_video.json
ruby .claude/skills/analyze-video/prepare_visual_script.rb libraries/[library]/transcripts/visual_video.json
2. Extract Frames (Binary Search)
Create frame directory: mkdir -p tmp/frames/[video_name]
Videos ≤30s: Extract one frame at 2s
Videos >30s: Extract start (2s), middle (duration/2), end (duration-2s)
ffmpeg -ss 00:00:02 -i video.mov -vframes 1 -vf "scale=1280:-1" tmp/frames/[video_name]/start.jpg
Subdivide when: Footage start, middle and end have different subjects, setting or angle changes
Stop when: The footage no longer seems to be changing or only has minor changes
Never sample more frequently than once per 30 seconds
3. Add Visual Descriptions
Read the visual video json file that you created earlier.
Read the JPG frames from tmp/frames/[video_name]/ using Read tool, then Edit visual_video.json:
Do these incrementally. You don't need to create a program or script to do this, just incrementally edit the json whenever you read new frames.
Dialogue segments - add visual field:
{
"start": 2.917,
"end": 7.586,
"text": "Hey, good afternoon everybody.",
"visual": "Man in red shirt speaking to camera in medium shot. Home office with bookshelf. Natural lighting.",
"words": [...]
}
B-roll segments - insert new entries:
{
"start": 35.474,
"end": 56.162,
"text": "",
"visual": "Green bicycle parked in front of building. Urban street with trees.",
"b_roll": true,
"words": []
}
Guidelines:
- Descriptions should be 3 sentences max.
- First segment: detailed (subject, setting, shot type, lighting, camera style)
- Continuing shots: brief if similar, otherwise can be up to 3 sentences if drastically different.
4. Cleanup & Return
rm -rf tmp/frames/[video_name]
Return structured response:
✓ [video_filename.mov] analyzed successfully
Visual transcript: libraries/[library]/transcripts/visual_video.json
Video path: /full/path/to/video_filename.mov
DO NOT update library.yaml - parent agent handles this to avoid race conditions in parallel execution.
