transcription-analysis

# Skill: Transcription Analysis > Use this skill for tasks involving transcription providers, caption streaming, or the audio pipeline. > Reference: `portal/transcription/`, [TRANSCRIPTION_MAP.md](../../context/TRANSCRIPTION_MAP.md). --- ## Audio Pipeline (end-to-end) ``` Interpreter browser └── getUserMedia (mic, 16-bit PCM via WebRTC/WHIP) └── MediaMTX (RTSP port 8554, path: {event_slug}/{language_code}) └── ffmpeg (spawned by worker.py) -rtsp_transport tcp -i rtsp://mediamtx:8554/{path} -f s16le -acodec pcm_s16le -ar 16000 -ac 1 - └── TranscriptionProvider.run_stream() └── 3-second PCM chunks └── process_chunk() → text └── CaptionAggregator └── broadcast_transcription() ├── /ws/booth/{booth_id} (interpreters) └── /ws/captions/{booth_id} (listeners) ``` --- ## Worker State - `active_workers: dict[str, dict]` — keyed by `booth_id`; value has `{task, provider, stderr_task}` - `active_processes: dict[str, asyncio.subprocess.Process]` — the ffmpeg process per booth - Both protected by `active_workers_lock: asyncio.Lock` - `MAX_TOTAL_WORKERS = 10` — process-global hard limit --- ## Adding a New Provider (checklist) 1. Create `portal/transcription/providers/{name}.py` 2. Implement class inheriting `TranscriptionProvider` from `providers/base.py` 3. Implement `process_chunk(chunk, language_code, model_variant, config, booth_state) -> str` 4. For streaming (WebSocket-based): override `run_stream(process, language_code, model_variant, config, broadcast_callback, booth_id)` — see `deepgram.py` and `openai.py` as references 5. Add `ProviderEnum.{NAME} = "{name}"` in `constants.py` 6. Add `{name}: {allowed_models_set}` to `ALLOWED_MODELS` 7. Add to `PROVIDERS` dict in `worker.py` 8. If API key needed: add encrypted column to `Event` model (migration 008 pattern) + update `get_api_key` key_map in `base.py` 9. Update `admin_event_api_settings_post` in `portal/routers/admin/settings.py` to handle new key 10. Update `templates/admin/api_settings.html` --- ## CaptionAggregator Usage Providers call aggregator methods, not `broadcast_callback` directly: ```python aggregator = CaptionAggregator(broadcast_callback) await aggregator.handle_partial(booth_id, partial_text) # streaming partial await aggregator.handle_final(booth_id, final_text) # completed utterance await aggregator.handle_chunk(booth_id, whisper_chunk) # Whisper-style chunks await aggregator.handle_clear(booth_id) # silence endpoint ``` **Forced finalization:** if an utterance exceeds 50 words or 15 seconds, it auto-finalizes regardless of provider signal. This prevents endlessly growing partials. --- ## Local Provider Details - **Model loading:** `get_model(model_size)` — thread-safe lazy loading. Model is loaded once and reused. - **Overlap buffer:** 1 second (32 000 bytes) of previous chunk is prepended to current chunk → 4-second effective window per inference call. - **Thread pool:** `asyncio.to_thread(self._run_inference, ...)` — each chunk is inferred in a thread. - **VAD filter:** `vad_filter=True` in faster-whisper — reduces empty segment noise. - **Model eviction:** `eviction_loop` checks every 15 min; evicts model if no active booths and last used > 1 hour ago. --- ## OpenAI Provider Details - `whisper-1`: uses REST `POST /v1/audio/transcriptions` (WAV conversion via `pcm_to_wav`) - `gpt-4o-realtime-*`: uses `wss://api.openai.com/v1/realtime` WebSocket; streams raw PCM - Retry: `tenacity` with exponential backoff (2–10s, 3 attempts) on 429/5xx - HTTP client: `portal.transcription.shared_http_client` — created in `lifespan` in `fastapi_app.py` --- ## Deepgram Provider Details - Connects to `wss://api.deepgram.com/v1/listen?model=nova-2&...&interim_results=true` - Runs concurrent `sender` (pipes PCM) and `receiver` (gets transcripts) tasks - Handles `KeepAlive` (every 5s timeout to prevent WS close) - `speech_final=True` or `is_final=True` → `handle_final`; otherwise → `handle_partial` --- ## NVIDIA Provider Details - Uses Riva gRPC (requires `nvidia-riva-client` optional dependency: `uv add eventyay-interpretation-portal[nvidia]`) - Models: `parakeet-rnnt`, `parakeet-ctc` - Requires `NVIDIA_FUNCTION_ID` setting in `portal/config.py` --- ## Transcription Trigger (Admin → Worker) 1. Admin sets `transcription_enabled=True`, `provider`, `model` via booth detail form. 2. `POST /admin/.../booths/{id}/transcription-settings` handler: - Validates provider + model combination. - Verifies API key exists if non-local provider. - Saves settings to DB. - If booth is currently live (has `active_interpreter_id`): stops old worker, starts new worker. 3. On interpreter Go Live: worker is started if `transcription_enabled=True` on the DBBooth. - Look for this logic in `portal/routers/api.py` (search for `start_transcription_worker`). --- ## Debugging Transcription 1. Check `active_workers` dict for the booth_id. 2. Check ffmpeg process is alive: `active_processes[booth_id].returncode is None`. 3. Check MediaMTX RTSP is reachable: `rtsp://mediamtx:8554/{event_slug}/{language_code}`. 4. Check API key: `get_api_key(event, ProviderEnum.X)` — returns `None` if not set. 5. Check `MAX_TOTAL_WORKERS` not exceeded. 6. Check `event.transcription_api_enabled` is `True` for external providers.