| license | Apache-2.0 |
| name | audio-transcription-pipeline |
| description | Build audio transcription pipelines with Whisper, Deepgram, and AssemblyAI including speaker diarization and real-time streaming. Activate on: transcription, speech-to-text, diarization, audio processing, meeting transcripts. NOT for: text-to-speech synthesis (voice-audio-engineer), music generation (ai-engineer). |
| allowed-tools | Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*) |
| category | Video & Audio |
| tags | ["transcription","speech-to-text","whisper","diarization","audio"] |
| pairs-with | [{"skill":"voice-audio-engineer","reason":"Audio engineer handles TTS and audio processing; this skill handles STT"},{"skill":"video-frame-extraction-analysis","reason":"Transcription syncs with video frames for multimedia indexing"},{"skill":"data-pipeline-engineer","reason":"Large-scale batch transcription needs pipeline orchestration"}] |
Audio Transcription Pipeline
Build production speech-to-text pipelines with Whisper, Deepgram, and AssemblyAI for batch and real-time transcription with speaker diarization.
Decision Points
Engine Selection Decision Tree
If requirements include:
├─ Real-time streaming required?
│ ├─ Yes: Use Deepgram Nova-3 WebSocket (fastest)
│ └─ No: Continue to accuracy requirements
├─ Highest accuracy needed + have GPU?
│ ├─ Yes: Use Whisper large-v3 with faster-whisper
│ └─ No: Continue to cost analysis
├─ Budget < $0.0059/min AND have compute?
│ ├─ Yes: Use local Whisper
│ └─ No: Use AssemblyAI Universal-2 (best API diarization)
Batch vs Stream Processing
If audio characteristics:
├─ Duration > 30 minutes?
│ ├─ Yes: Use batch with chunking (split at silence)
│ └─ No: Continue to latency check
├─ Need results in < 10 seconds?
│ ├─ Yes: Use streaming (Deepgram/Whisper.cpp)
│ └─ No: Use batch for better accuracy
VAD (Voice Activity Detection) Usage
If content type:
├─ Live conversation/meeting?
│ ├─ Yes: Enable VAD (saves 40-60% compute on silence)
│ └─ No: Continue to content check
├─ Lecture/presentation with pauses?
│ ├─ Yes: Use conservative VAD (min_silence_duration_ms: 1000)
│ └─ No: Skip VAD for dense speech (audiobooks, etc.)
Model Selection by Domain
If language/accent:
├─ Non-English or heavy accent?
│ ├─ Yes: Use Whisper large-v3 (best multilingual)
│ └─ No: Continue to speed check
├─ Need < 1 second latency?
│ ├─ Yes: Use Deepgram Nova-3 streaming
│ └─ No: Use AssemblyAI for business/medical terminology
Failure Modes
Hallucination on Silence
Detection: Transcripts show repeated phrases during quiet sections or phantom music descriptions
Diagnosis: VAD disabled or threshold too low, model generating text from background noise
Fix: Enable VAD with min_silence_duration_ms: 500, use --no_speech_threshold 0.6 in Whisper
Diarization Collapse at >10 Speakers
Detection: All speakers labeled as "Speaker 1" after 10-15 minutes, or random speaker switching mid-sentence
Diagnosis: Speaker embedding model saturated, overlap confusion in crowded audio
Fix: Pre-segment by silence, use AssemblyAI dual_channel if stereo, limit to 8 active speakers max
Timestamp Sync Drift with Video
Detection: Subtitles appear 2-5 seconds before/after corresponding video frames
Diagnosis: Audio preprocessing changed duration, or VAD removed segments without timestamp adjustment
: Use in preprocessing, sync with original audio timecode, validate against known speech events
