video-watching

name	video-watching
description	Watch a video the way a human would — turn it into a time-mapped transcript with the on-screen visuals (slides, UI, diagrams, text) described inline, then reason about it. Triggers: "watch this video", "şu videoyu izle", "videoyu analiz et", "transkript çıkar", "video transcript", "bu videoda ne var", or any time the user hands over a video file/link. Produces a curated <video>.transcript.md NEXT TO the source video, then continues per use-case (marketing teardown, FTE analysis, knowledge capture, …).
alwaysApply	false
ruleType	Tool Workflow
version	1.0

Video Watching

Turn a video into something an AI can fully reason about: a time-mapped transcript with what's on screen described inline at the same timestamps. The output is one self-contained file dropped next to the source video, so any model (our cloud, this session, anything) can read the whole video without the binary.

The clever part: transcript first, then YOU decide which frames to look at. You don't blindly OCR every frame. You read the transcript, see where the speaker references something visual ("as you can see here", "this screen", "the chart"), look up that moment in frames.json, and open only those frames.

The loop (do these in order)

0. Get the video

Ask for the path or link if not given. Local files work today. Remote links (YouTube etc.) need yt-dlp (brew install yt-dlp) — the engine auto-downloads if it's installed.

1. Run the engine (mechanical — one command)

.claude/skills/video-watching/scripts/transcribe.sh "/abs/path/to/video.mov"
#   language auto-detects — no need to pass it. Override only if auto mislabels a
#   short/ambiguous clip: ./transcribe.sh "/abs/path/clip.mp4" tr

Pick the mode for the kind of video — this matters most for app/UI recordings:

• Talking-head / lecture / ad creative → the default is right. Scene-detect + a 10s gap-fill catches the visuals.
• App screen-recording / onboarding funnel / UI walkthrough → add --mode ui. App screens linger 3–5s and change by text only (a questionnaire step, a paywall) — they don't move enough to trip scene-detect, so the 10s default silently drops most of them. --mode ui samples every ~2.5s so each screen lands. If you only need the screens (no narration), add --frames-only to skip whisper:

  ./scripts/transcribe.sh "/abs/path/onboarding.mp4" --mode ui --frames-only --contact-sheet
  

This writes everything into <video_dir>/<slug>.media/:

• transcript.srt / .json / .txt — timestamped transcript (large-v3-turbo) — skipped with --frames-only
• frames/anchor_first.jpg / anchor_last.jpg — first + last frame, always captured (short cut-heavy creatives carry the hook and CTA here; scene-detect misses both)
• frames/frame_*.jpg — frames selected in one time-based pass: a scene change fired (slide/UI transition) or MAX_GAP seconds elapsed since the last frame, whichever comes first. The gap rule is wall-clock based (prev_selected_t), so it works on variable-frame-rate screen recordings where frame-number sampling breaks, and it guarantees every static stretch (app demos, slides, CTAs) gets a frame.
• frames/contact_sheet.jpg — tiled montage of all frames, only with --contact-sheet
• frames.json — the index you read: [{file, t, at, type}], sorted by time, near-duplicate timestamps collapsed (anchors always kept). type is scene (a picture change fired), gap (a periodic fill at the MAX_GAP cadence), or anchor.

The engine prints a coverage check at the end: longest unsampled gap = Xs. If it warns the gap is >2× MAX_GAP, frames are likely missing — re-run denser (--max-gap lower, or --mode ui) before any expensive deep-analysis pass.

audio.wav is auto-deleted after transcription (it's a ~1.9MB/min whisper-only intermediate). The whole *.media/ dir is gitignored — it stays next to the video as scratch but never enters the repo. Only <slug>.transcript.md is the keeper; delete a .media/ folder anytime to reclaim disk (re-running regenerates it).

2. Read the transcript

Read transcript.srt. It carries the spoken content with [mm:ss] timing. Form a first-pass understanding of structure and topic.

3. Decide which frames you NEED — then view them

Read frames.json. For each moment where understanding depends on the visual — the speaker points at something, a slide/UI/diagram/number is on screen, or the transcript is ambiguous without the picture — pick the frame whose at is closest to that moment and open it with the Read tool (it renders images). Don't open all of them; open the ones that carry information. Talking-head stretches with nothing on screen need no frame.

Heuristic: the two anchor frames (first/last) almost always matter — the hook and the CTA. Every scene frame is a candidate (the picture changed for a reason). gap frames cover static stretches scene-detect skipped — often the most informative part (an app demo or onboarding screen that doesn't "move"), so check them. In --mode ui runs most frames are gap — that's expected and you generally want to look at all of them, one per screen.

Need a frame the index doesn't have? Grab it on demand. Scene-detect fires on motion, not on meaning — on fast-cut video the most informative moment often sits between the indexed frames. When the VO points at something ("look at this screen", a number, a result) and no indexed frame lands there, fetch that exact second yourself:

ffmpeg -y -loglevel error -ss <seconds> -i "<video>" -frames:v 1 -q:v 3 \
  "<slug>.media/frames/grab_<mmss>.jpg"

This is the whole point of transcript-first: the frame set is yours to extend, not a fixed dump you're stuck with.

4. Write the curated transcript NEXT TO the video

Write <video_dir>/<slug>.transcript.md — the deliverable. One timeline, voice and visuals interleaved, every line timestamped:

# <Video Title>
source: <path or URL> · duration: <mm:ss> · transcribed: large-v3-turbo

## Summary
2–4 sentences: what this video is and what it's for.

## Timeline
- **[00:00]** <what's said>
- **[00:06]** 🖼 <what's on screen — slide title, UI state, chart, on-screen text>
- **[00:12]** <what's said> … 🖼 <visual if relevant>
...

## Visual notes
- **[00:06]** Slide: "Funnel → Billing → Gateway → PSP" (4 layers)
- ...

Rules: never invent content not in transcript or frames; mark anything uncertain; keep the source path in the header so the file stands alone.

5. Hand off to the use-case

Once the .transcript.md exists you know the video cold. Tell the user it's ready and ask what they want from it — the answer is domain-specific. Triage to the relevant dreamcontext skill (per the skill-triage rule) and load it:

• Marketing video → teardown, hook/CTA analysis, "we could do X" (→ growth / meta-marketing skills)
• First-time-experience / app demo → reconstruct the FTE flow, friction points (→ design / onboarding-design)
• Knowledge / training → if it should persist for the project, hand the transcript to dreamcontext knowledge so it becomes durable context Don't guess the use-case — let the user direct it.

UI teardown (no transcript needed). When the goal is purely the app flow — e.g. tearing down a competitor's onboarding funnel — run --frames-only --mode ui and skip the .transcript.md entirely. The deliverable becomes a curated screen list (one entry per onboarding step, in order, from frames.json), which feeds a board (excalidraw) or an onboarding-design analysis directly. Use --contact-sheet to eyeball coverage first, and trust the coverage warning — under-sampling here is exactly what makes a teardown wrongly report "screen X wasn't shown".

What this skill is NOT

• Not a knowledge-base writer or ingestion pipeline. This skill produces the transcript artifact; persisting it (chunking, embedding, ingest into a project's long-term memory) is a separate, downstream concern — keep that boundary clean.
• Not an auto-summarizer that skips the frames. The visual pass is the point — that's what separates "watched the video" from "read the subtitles".

Setup (once per machine)

brew install whisper-cpp ffmpeg          # yt-dlp too, only for remote links
# model: the engine prefers large-v3-turbo, falls back to large-v3 then medium.
# pull turbo if missing:
#   curl -L -o ~/.cache/whisper.cpp/models/ggml-large-v3-turbo.bin \
#     https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo.bin

build_frame_index.py is stdlib-only — no venv needed. Language auto-detects (whisper); only pass a code (tr, en) to force it on a short/ambiguous clip.

Files

.claude/skills/video-watching/
├── SKILL.md                    ← you are here
└── scripts/
    ├── transcribe.sh           ← video → transcript + frames + frames.json (the engine)
    └── build_frame_index.py    ← frames/*.jpg → frames.json (pts_time index + coverage check)

Flags & tuning

Flags (each also settable as an env var, e.g. MODE=ui):

• --mode ui|lecture — sampling preset. ui = MAX_GAP 2.5s (app screens); lecture (default) = 10s (talking-head). Explicit --max-gap/--scene-threshold always win.
• --frames-only — skip whisper; extract frames only (UI/UX teardowns).
• --max-gap N — guarantee a frame at least every N seconds.
• --scene-threshold N — scene-change sensitivity, lower = more frames.
• --contact-sheet — also emit frames/contact_sheet.jpg (coverage at a glance).
• --lang CODE — force the transcript language (default: auto-detect).

Common adjustments:

• Onboarding/UI recording under-sampled? --mode ui (or push further: --max-gap 1.5).
• Too few frames on a slide-heavy talk? --scene-threshold 0.1.
• Long static lecture over-sampled? --max-gap 20.
• Capturing too many near-identical frames? DEDUPE_SEC=1.0 ./transcribe.sh ….
• Force a model: WHISPER_MODEL=/abs/ggml-large-v3.bin ./transcribe.sh ….