media-understanding

name	media-understanding
description	Understand audio and video content. Use when user asks to transcribe, analyze, summarize, or extract information from audio/video files or YouTube URLs. Routes automatically: audio files use local whisper-ctranslate2 (free, fast), video files use Gemini API (visual+audio understanding).

Media Understanding

Setup

Prerequisites

# Install whisper-ctranslate2 for audio transcription
uv tool install whisper-ctranslate2

# Verify installation
whisper-ctranslate2 --help

GPU Acceleration (Optional)

NVIDIA (CUDA):

# Install CUDA-enabled ctranslate2
pip install ctranslate2 --force-reinstall --extra-index-url https://download.pytorch.org/whl/cu121

AMD (ROCm):

# ROCm 6.x only (7.x has breaking changes)
pip install ctranslate2 --force-reinstall --extra-index-url https://download.pytorch.org/whl/rocm6.0

CPU (default): No extra setup needed. Use --compute_type int8 for best performance.

API Keys

# Required for video understanding
export GEMINI_API_KEY="your-key-here"

Audio Files → whisper-ctranslate2 (local)

For mp3, wav, m4a, flac, ogg, aac files:

whisper-ctranslate2 "path/to/audio.mp3" --output_dir /tmp --output_format srt --model large-v3

CPU-optimized (recommended for Intel/no GPU):

whisper-ctranslate2 "path/to/audio.mp3" --output_dir /tmp --output_format srt --model large-v3-turbo --compute_type int8

Options

Option	Description
--output_dir DIR	Output directory for transcription files
--output_format FMT	txt, srt, vtt, json, all (default: all)
--model SIZE	tiny, base, small, medium, large-v3, large-v3-turbo (default: small)
--language LANG	Force language (auto-detected by default)
--task transcribe	Transcribe in original language (default)
--task translate	Translate to English
--word_timestamps true	Include word-level timing
--device cpu/cuda	Force CPU or GPU (default: auto)
--compute_type TYPE	int8 (CPU), float16 (GPU), float32
--vad_filter true	Filter non-speech audio (good for music/noise)

Model Selection

Model	Size	Speed	Accuracy	Use Case
tiny	39M	Fastest	Low	Quick drafts
base	74M	Fast	Medium	General use
small	244M	Medium	Good	Balanced
medium	769M	Slow	High	Quality focus
large-v3	1.5G	Slowest	Highest	Best accuracy
large-v3-turbo	809M	Fast	High	Best for CPU
distil-large-v3	756M	Fast	High	Distilled, English-only

Output: Transcription file(s) in output directory.

Video Files → Gemini (visual + audio)

For mp4, mov, webm, avi, mkv files or YouTube URLs:

uv run ~/.claude/skills/media-understanding/scripts/understand_video.py \
  --source "path/to/video.mp4" \
  --prompt "Describe what happens in this video"

Options

Option	Description
--fast	Use faster flash model
--fps N	Frame rate sampling (default: 1 fps)
--start N	Start time in seconds
--end N	End time in seconds

Example Prompts

• "Summarize this video in 3 bullet points"
• "Transcribe all spoken dialogue with timestamps"
• "What text appears on screen?"
• "Describe the main actions and events"

Fresh System Installation

# Clone the skill
git clone https://github.com/ferdousbhai/media-understanding.git ~/Projects/media-understanding

# Create skills directory and symlink
mkdir -p ~/.claude/skills
ln -sf ~/Projects/media-understanding ~/.claude/skills/media-understanding

# Install dependencies
uv tool install whisper-ctranslate2

# Set API key (add to ~/.bashrc or ~/.zshrc)
echo 'export GEMINI_API_KEY="your-key-here"' >> ~/.bashrc