speech-expert

name	speech-expert
description	Expert guide for speech-mcp - TTS, local/streaming STT, barge-in, wake word, voice command bus, and provider selection.

Speech-MCP Expert

speech-mcp is a multi-provider speech gateway: text-to-speech (TTS), speech-to-text (STT), realtime voice, wake word, and a fleet voice command bus. It runs as an MCP server (stdio or HTTP) with a React webapp.

When to use which STT

Need	Use	Why
Batch/file transcription	transcribe_audio_file (provider funasr)	FunASR MLT-Nano: offline, multilingual, VAD+punctuation+diarization
Live streaming with partials	streaming_stt (sherpa-onnx)	ja/en/de, CPU, emits partials + endpoint
Barge-in / interruption	barge_in_feed	Completed utterance = interrupt signal
Cloud quality, realtime conversation	start_evi_session / Gemini Live	Hume EVI or gemini_live
Tiny chunks / stateless	transcribe_stream_chunk	base64 PCM per fragment

Guidance: default to funasr for accuracy + offline; use streaming_stt when you need text as it is spoken (voice agents, live captions); use barge_in_feed to let the user interrupt the assistant (no echo cancellation - gate mic feeding while it speaks).

TTS providers

text_to_speech(provider=...) - windows (SAPI5, always works), gemini (highest cloud quality, voice=Kore etc.), hume (emotional/expressive), elevenlabs (needs voice_id from manage_voice_clones), gemma (local, SAPI fallback). text_to_dialogue = multi-voice ElevenLabs conversation.

Wake word + voice command bus

• configure_local_wake_word(action="start|stop|status") - openWakeWord, offline.
• With FLEET_VOICE_DELEGATE=1 the listener routes wake -> STT -> POST fleet-agent /api/voice/intent. SHERPA_ASR_ENABLED=1 switches capture to streaming (no fixed window) with a follow-up turn.

REST surface

/api/v1/health (provider booleans), /api/v1/diagnostics, /api/tools, /api/skills, /api/v1/tts, /api/v1/voices, /api/v1/transcribe, /api/v1/wake_word, /api/v1/utility, /api/v1/stop, /api/v1/shutdown.

Config essentials

• FunASR: FUNASR_ENABLED=true, FUNASR_MODEL=FunAudioLLM/Fun-ASR-MLT-Nano-2512 (31 langs incl. en/ja/de), FUNASR_DEVICE=cuda:0|cpu. Requires uv sync --extra funasr.
• Streaming: SHERPA_ASR_ENABLED=true, SHERPA_ASR_LANG=en|ja|de, SHERPA_BARGE_IN=true. Requires uv sync --extra sherpa + just sherpa-download.
• Cloud keys: GOOGLE_API_KEY, HUME_API_KEY, ELEVENLABS_API_KEY.

Troubleshooting quick hits

• Health funasr:false -> funasr extra not installed (uv sync --extra funasr) or model needs trust_remote_code (already set). Device cpu vs cuda:0 mismatch.
• streaming_stt unavailable -> SHERPA_ASR_ENABLED off or model not downloaded.
• No sound -> providers map in /api/v1/health; try windows provider first.
• REST auth -> SPEECH_MCP_AUTH_TOKEN must be sent as X-Speech-MCP-Auth.

Full docs: docs/STREAMING_ASR.md, docs/providers/funasr.md, docs/TROUBLESHOOTING.md.