This skill performs deep analysis of YouTube videos through **both information channels** Multimodal YouTube video analysis through both audio (transcript) and visual (frame extraction + image analysis) channels. Especially powerful for HowTo videos, tutorials, demos, and explainer videos where what is SHOWN (screenshots, UI demos, diagrams, code, physical actions) is just as important as what is SAID. Use this skill whenever a user wants to analyze, summarize, or create step-by-step guides from YouTube videos, or when they share a YouTube URL and want to understand what happens in the video. Triggers on requests like "Analyze this YouTube video", "Create a step-by-step guide from this video", "What does this video show?", "Summarize this tutorial", or any YouTube URL shared with analysis intent.
Instrucciones de origen · Vista previa de solo lectura
name
youtube-knowledge-extractor
description
This skill performs deep analysis of YouTube videos through **both information channels** Multimodal YouTube video analysis through both audio (transcript) and visual (frame extraction + image analysis) channels. Especially powerful for HowTo videos, tutorials, demos, and explainer videos where what is SHOWN (screenshots, UI demos, diagrams, code, physical actions) is just as important as what is SAID. Use this skill whenever a user wants to analyze, summarize, or create step-by-step guides from YouTube videos, or when they share a YouTube URL and want to understand what happens in the video. Triggers on requests like "Analyze this YouTube video", "Create a step-by-step guide from this video", "What does this video show?", "Summarize this tutorial", or any YouTube URL shared with analysis intent.
This skill performs deep analysis of YouTube videos through both information channels:
Audio channel: Transcript with timestamps (what is SAID)
Visual channel: Frame extraction + image analysis (what is SHOWN)
Most YouTube skills only extract transcripts. This skill closes the gap by synchronizing visual frames with spoken content, enabling accurate step-by-step guides where "click the blue button" is matched with the actual screenshot showing which button.
Workflow Overview
YouTube URL
|
+---> 1. Get metadata (title, duration, video ID)
|
+---> 2. Extract transcript (yt-dlp --dump-json + curl)
| -> Timestamped segments
|
+---> 3. Extract frames (yt-dlp + ffmpeg)
| -> Keyframes at strategic intervals
|
+---> 4. Synchronize frames <-> transcript
| -> Match frames to spoken content by timestamp
|
+---> 5. Multimodal analysis
-> Read each frame image, combine with transcript
-> Generate structured output
yt-dlp --print title --print duration --printid"$VIDEO_URL" 2>/dev/null
This returns three lines: title, duration in seconds, video ID. Store these for later use.
Step 3: Extract Transcript
IMPORTANT: Direct subtitle download via --write-sub frequently hits YouTube rate limits (HTTP 429).
Use the reliable two-step method below instead.
Step 3a: Get subtitle URL from video JSON
yt-dlp --dump-json "$VIDEO_URL" 2>/dev/null | python3 -c "
import json, sys
data = json.load(sys.stdin)
auto = data.get('automatic_captions', {})
subs = data.get('subtitles', {})
# Priority: manual subs > auto subs. Prefer user's language, fallback chain.
for source in [subs, auto]:
for lang in ['en', 'de', 'en-orig', 'fr', 'es']:
if lang in source:
for fmt in source[lang]:
if fmt.get('ext') == 'json3':
print(fmt['url'])
sys.exit(0)
# Fallback: take first available auto-caption, get json3 URL
for lang in sorted(auto.keys()):
for fmt in auto[lang]:
if fmt.get('ext') == 'json3':
url = fmt['url']
# Remove translation param to get original language
import re
url = re.sub(r'&tlang=[^&]+', '', url)
print(url)
sys.exit(0)
print('NO_SUBS', file=sys.stderr)
sys.exit(1)
" > "$WORK_DIR/sub_url.txt"
head -c 20 "$WORK_DIR/transcript.json3"# Should start with { — if it starts with <html, retry after 10s sleep
Step 3c: Parse json3 into readable timestamped segments
python3 -c "
import json
with open('$WORK_DIR/transcript.json3') as f:
data = json.load(f)
for event in data.get('events', []):
segs = event.get('segs', [])
if not segs:
continue
start_ms = event.get('tStartMs', 0)
duration_ms = event.get('dDurationMs', 0)
text = ''.join(s.get('utf8', '') for s in segs).strip()
if not text or text == '\n':
continue
s = start_ms / 1000
e = (start_ms + duration_ms) / 1000
print(f'[{int(s//60):02d}:{int(s%60):02d} - {int(e//60):02d}:{int(e%60):02d}] {text}')
" > "$WORK_DIR/transcript.txt"
Read $WORK_DIR/transcript.txt to get the full transcript with timestamps.
Fallback: No transcript available
If no subtitles exist at all, inform the user and proceed with visual-only analysis.
Step 4: Download Video and Extract Frames
Step 4a: Download video (720p is sufficient for frame analysis)
Physical actions: Hand positions, tool usage (for physical HowTos)
Changes: What changed compared to the previous frame?
Step 6b: Synthesize both channels
For each key moment, combine audio and visual:
Segment [TIMESTAMP]:
SAID: "Click the blue button in the top right"
SHOWN: Settings page screenshot, blue "Save" button highlighted
in top-right corner, cursor pointing at it
SYNTHESIS: -> On the Settings page, click the blue "Save" button
in the top-right corner
Step 6c: Identify visual-only information
Flag moments where the visual channel provides information NOT present in audio:
Specific button names, menu paths, exact UI locations
Code that is shown but not read aloud
Error messages visible on screen
Before/after comparisons
Output Formats
Generate the appropriate format based on the user's request:
Format A: Step-by-Step Guide (most common)
# [Video Title] — Guide## Step 1: [Action] (00:15)
[Description based on transcript + frame analysis]
> Visual: [What the screen/image shows at this point]## Step 2: [Action] (00:42)
[...]
Format B: Comprehensive Summary with Visual Anchors
Physical HowTos: Use tighter frame intervals (10-15s) — movements are subtler
Read the transcript first: Identify "interesting timestamps" before extracting frames. Look for phrases like "as you can see here", "let me show you", "on the screen" — these signal important visual moments
Context-aware frame analysis: When analyzing a frame, always provide the transcript context. The speaker often explains what's about to be shown
Batch frame reading: Read frames in batches of 8-10 to maintain context across sequential frames and detect visual changes
Always extract both channels in parallel: Start the video download while processing the transcript to save time
