01-shorts-analyzer

name	01-shorts-analyzer
description	YouTube Shorts 영상을 60fps(1/60초) 단위로 프레임 분석하여 구조화된 데이터로 변환. Use when "영상 분석", "프레임 분석", "쇼츠 분석", "video analysis", "frame analysis", "영상 따라 만들기", "영상 재현".

Shorts Analyzer

Decompose YouTube Shorts into 60fps frames and extract layout, text, color, and animation into structured analysis documents.

Pipeline: Download -> Audio analysis -> Parallel Vision analysis -> Subtitle cross-validation

Recognition Pattern

• Analyzing YouTube Shorts channel video styles
• Recreating/cloning videos with Remotion
• Reverse-engineering video design systems (colors, fonts, layout)
• Extracting common patterns across multiple videos

Required Tools

yt-dlp (download) | ffmpeg (60fps frame extraction) | Vision Agent/Sonnet (frame analysis) | Task() (parallel orchestration)

Execution

Phase 0: Preflight

Ask user before Phase 1:

AskUserQuestion:
  "화면 레이아웃이 영역별로 나눠져 있나요?"
  - 예 (고정 영역 분할) -> Phase 3 includes layout ratio analysis
  - 아니오 (전체 화면)   -> Phase 3 skips layout analysis

Checklist

• Layout split confirmed with user
• Result reflected in Phase 3 analysis scope

Phase 1: Download & Frame Extraction

Input: https://www.youtube.com/shorts/[videoId]

mkdir -p [videoId]
yt-dlp -o "[videoId]/video.%(ext)s" "https://www.youtube.com/shorts/[videoId]"
ffmpeg -i "[videoId]/video.mp4" -vn -acodec libmp3lame -q:a 2 "[videoId]/01-audio.mp3"
mkdir -p [videoId]/frames
ffmpeg -i "[videoId]/video.mp4" -vf "fps=60" -q:v 2 "[videoId]/frames/frame_%04d.jpg"

Output: [videoId]/ with video.mp4, 01-audio.mp3, frames/frame_0001.jpg ~ frame_NNNN.jpg

CRITICAL: Always extract audio. Missing audio = silent video in recreator.

Checklist

• [videoId]/video.mp4 exists
• [videoId]/01-audio.mp3 exists
• [videoId]/frames/frame_0001.jpg ~ frame_NNNN.jpg exist
• Frame count = video length x 60 (+/-1)

Phase 2: Audio Analysis (vocals, bgm, sfx)

Analyze audio before Vision. Demucs separates vocals/bgm/sfx; per-frame RMS enables cross-validation with Vision animation timing.

Input: [videoId]/01-audio.mp3

2-1. Environment Setup (first time only)

cd .agents/skills/01-shorts-analyzer && bash scripts/setup.sh

Creates .venv via mise python@lts + uv with demucs, torch, torchaudio, numpy.

2-2. STT

.agents/skills/01-shorts-analyzer/.venv/bin/whisper [videoId]/01-audio.mp3 --model large --language ko --output_format json --output_dir [videoId]

Local Whisper CLI, large model (~2.9GB), --language ko for best Korean accuracy. Output: [videoId]/01-audio.json

2-3. Demucs Separation + RMS Analysis

.agents/skills/01-shorts-analyzer/.venv/bin/python \
  .agents/skills/01-shorts-analyzer/scripts/analyze_audio.py \
  [videoId]/01-audio.mp3 --fps 60 \
  --output [videoId]/01-audio-demucs/demucs-result.json

Processing: Demucs htdemucs -> source separation -> per-frame RMS (1/60s = 735 samples @44.1kHz) -> SFX peak detection (>3dB & >-20dB) -> VO silence detection.

Output [videoId]/01-audio-demucs/demucs-result.json key fields (full schema: scripts/analyze_audio.py):

• rms_per_frame -- mix/vocals/other per-frame RMS dB arrays
• sfx_peaks -- other track volume spikes (frame, time_sec, delta_db)
• vo_silences -- vocal silence ranges (start/end frame+sec, duration)

vocals.wav = VO timing, other.wav = BGM+SFX mixed (peaks = SFX estimate). Full BGM/SFX separation not possible.

Checklist

• [videoId]/01-audio.json (STT) exists
• [videoId]/01-audio-demucs/demucs-result.json exists
• Contains rms_per_frame, sfx_peaks, vo_silences
• Contains summary.bpm, summary.instruments, summary.genres (PANNs)
• total_frames matches Phase 1 frame count

Phase 3: Parallel Vision Analysis

Chunk Division

100-frame (~1.7s) chunks with 50-frame overlap on each side:

Total: 2580 frames (43s x 60fps), chunk=100, overlap=50/side
chunk  1: frame_0001~0150 (next overlap only)
chunk  2: frame_0051~0250 (both sides)  ...
chunk 26: frame_2451~2580 (prev overlap only)

Overlap rules: Overlap analysis defers to the chunk owning that range as primary. Ensures transition/animation continuity at boundaries.

Parallel Execution

Task(subagent_type="oh-my-claudecode:vision", model="sonnet",
  prompt="Analyze frames 0051~0250. Prev overlap: 0051~0100, next: 0201~0250. Body: 0101~0200.")
# x chunk count in parallel

Output format: -> See references/vision-prompt-template.md for tree structure and examples.

Output

[videoId]/01-vision-chunks/chunk_01.md ~ chunk_NN.md (per-chunk) -> merged into [videoId]/01-analysis.md

Merge rules: Transitions as independent items between Scenes | No transition-in/out under 배경 (only effect) | Deduplicate transitions at chunk boundaries

Checklist

• All chunk agents completed (no missing chunks)
• [videoId]/01-vision-chunks/chunk_01.md ~ chunk_NN.md exist
• [videoId]/01-analysis.md merged, overlap deduplicated
• All Scenes have 배경/텍스트/그래픽/오디오/메시지 (5 sections) — 그래픽 없으면 그래픽: 없음 명시
• All text entries have [frame NNNN~NNNN] ranges
• All 그래픽 entries have position, size, effect, [frame NNNN~NNNN] (미상 금지)
• Transitions placed as independent items between Scenes

Phase 4: Subtitle Cross-Validation (Vision × Whisper)

Compare Vision text (01-analysis.md) with Whisper STT (01-audio.json):

1. Vision: "text" [frame] per Scene | Whisper: segments[].text + start/end
2. Convert frame→time, match overlapping segments

Verdicts: Match | Partial (keywords same, particles differ) | Mismatch | Vision only | Whisper only

Mismatch handling: Vision (screen text) takes priority — reproduction target. Whisper = timing reference only. Vision-only is normal.

Output

Cross-validation results appended to [videoId]/01-analysis.md as final section.

Checklist

• Subtitle cross-validation complete (Vision × Whisper)
• All Scene frame ranges continuous (no gaps)