| name | media-understanding |
| description | Understand audio and video content. Use when user asks to transcribe, analyze, summarize, or extract information from audio/video files or YouTube URLs. Routes automatically: audio files use local whisper-ctranslate2 (free, fast), video files use Gemini API (visual+audio understanding). |
Media Understanding
Setup
Prerequisites
uv tool install whisper-ctranslate2
whisper-ctranslate2 --help
GPU Acceleration (Optional)
NVIDIA (CUDA):
pip install ctranslate2 --force-reinstall --extra-index-url https://download.pytorch.org/whl/cu121
AMD (ROCm):
pip install ctranslate2 --force-reinstall --extra-index-url https://download.pytorch.org/whl/rocm6.0
CPU (default): No extra setup needed. Use --compute_type int8 for best performance.
API Keys
export GEMINI_API_KEY="your-key-here"
Audio Files → whisper-ctranslate2 (local)
For mp3, wav, m4a, flac, ogg, aac files:
whisper-ctranslate2 "path/to/audio.mp3" --output_dir /tmp --output_format srt --model large-v3
CPU-optimized (recommended for Intel/no GPU):
whisper-ctranslate2 "path/to/audio.mp3" --output_dir /tmp --output_format srt --model large-v3-turbo --compute_type int8
Options
| Option | Description |
|---|
--output_dir DIR | Output directory for transcription files |
--output_format FMT | txt, srt, vtt, json, all (default: all) |
--model SIZE | tiny, base, small, medium, large-v3, large-v3-turbo (default: small) |
--language LANG | Force language (auto-detected by default) |
--task transcribe | Transcribe in original language (default) |
--task translate | Translate to English |
--word_timestamps true | Include word-level timing |
--device cpu/cuda | Force CPU or GPU (default: auto) |
--compute_type TYPE | int8 (CPU), float16 (GPU), float32 |
--vad_filter true | Filter non-speech audio (good for music/noise) |
Model Selection
| Model | Size | Speed | Accuracy | Use Case |
|---|
tiny | 39M | Fastest | Low | Quick drafts |
base | 74M | Fast | Medium | General use |
small | 244M | Medium | Good | Balanced |
medium | 769M | Slow | High | Quality focus |
large-v3 | 1.5G | Slowest | Highest | Best accuracy |
large-v3-turbo | 809M | Fast | High | Best for CPU |
distil-large-v3 | 756M | Fast | High | Distilled, English-only |
Output: Transcription file(s) in output directory.
Video Files → Gemini (visual + audio)
For mp4, mov, webm, avi, mkv files or YouTube URLs:
uv run ~/.claude/skills/media-understanding/scripts/understand_video.py \
--source "path/to/video.mp4" \
--prompt "Describe what happens in this video"
Options
| Option | Description |
|---|
--fast | Use faster flash model |
--fps N | Frame rate sampling (default: 1 fps) |
--start N | Start time in seconds |
--end N | End time in seconds |
Example Prompts
- "Summarize this video in 3 bullet points"
- "Transcribe all spoken dialogue with timestamps"
- "What text appears on screen?"
- "Describe the main actions and events"
Fresh System Installation
git clone https://github.com/ferdousbhai/media-understanding.git ~/Projects/media-understanding
mkdir -p ~/.claude/skills
ln -sf ~/Projects/media-understanding ~/.claude/skills/media-understanding
uv tool install whisper-ctranslate2
echo 'export GEMINI_API_KEY="your-key-here"' >> ~/.bashrc
