shiny-speech

# Shiny Speech Skill You are an expert in Shiny Speech, a library that provides cross-platform speech-to-text, text-to-speech, audio capture, and audio playback for .NET MAUI and Blazor WebAssembly with pluggable cloud providers. ## When to Use This Skill Invoke this skill when the user wants to: - Add speech-to-text (STT) or text-to-speech (TTS) to a .NET MAUI app - Capture audio from the device microphone - Play audio streams on the device - Use Azure AI Speech for cloud-based STT/TTS - Use ElevenLabs for cloud-based TTS - Implement a custom cloud speech provider - Configure speech recognition options (language, silence timeout, on-device preference, keywords) - Configure text-to-speech options (voice, rate, pitch, volume) - Add emotion / tone / expressive delivery to synthesized speech - Use ElevenLabs v3 audio tags (`[excited]`, `[whispers]`) or Typecast emotion presets - Strip or normalize bracketed audio tags so they aren't spoken aloud - List available TTS voices - Start/stop continuous speech recognition with event-based results - Implement listen-until-silence dictation - Implement wake word / keyword activation ("Hey Siri" style) - Implement keyword listening (listen until a specific keyword is detected) - Listen for keywords continuously as an async stream - Add speech-to-text or text-to-speech to a Blazor WebAssembly app - Use the Web Speech API via Shiny.Speech in the browser ## Library Overview **GitHub**: https://github.com/shinyorg/speech **NuGet Packages**: - `Shiny.Speech` — Core library with platform-native STT, TTS, audio capture, and playback - `Shiny.Speech.Cloud` — Cloud provider abstractions - `Shiny.Speech.Azure` — Azure AI Speech provider - `Shiny.Speech.ElevenLabs` — ElevenLabs TTS provider - `Shiny.Speech.Typecast` — Typecast TTS provider (TTS only, via the `typecast-csharp` SDK) - `Shiny.Audio.Linux` — Linux audio backend (PulseAudio/PipeWire with ALSA fallback). Required on Linux; nothing else registers audio services there - `Shiny.Speech.Linux.Whisper` — On-device, offline STT on Linux via Whisper/whisper.cpp. STT only (there is no Whisper TTS) **Namespace**: `Shiny.Speech` Shiny Speech provides: - Platform-native speech-to-text via `ISpeechToTextService` (iOS, Android, Windows, Browser/WASM) - Event-based recognition model — `ResultReceived`, `KeywordHeard`, `Error` events allow multiple subscribers - Start/Stop lifecycle — call `Start()` to begin listening, `Stop()` to end; `Start()` throws if already listening - Built-in keyword detection — set `Keywords` in `SpeechRecognitionOptions` and subscribe to `KeywordHeard` - Platform-native text-to-speech via `ITextToSpeechService` (iOS, Android, Windows, Browser/WASM) - Platform-native audio capture via `IAudioSource` (raw PCM 16kHz, 16-bit, mono — all platforms including browser) with an `InputLevelChanged` VU signal on every platform - Platform-native audio playback via `IAudioPlayer` — play a `Stream`, or a remote URL / local file path via `PlayAsync(string)` (platform resolves the source natively; browser uses HTML5 Audio) - Concurrent playback — clips overlap instead of interrupting each other; `StartAsync` returns an `IAudioPlayback` handle (`Id`, `Source`, `IsPlaying`, `Completion`, `StopAsync()`) that stops one clip on its own, `IAudioPlayer.Active` lists what is playing, and `IAudioPlayer.StopAsync()` stops everything - Live microphone monitor via `IAudioMonitor` — routes the mic to the current output in near-real-time (PA / "talk over a Bluetooth speaker"); `Start`/`Stop`, adjustable `Gain`, `InputLevelChanged` VU signal, `AudioMonitorOptions` (voice processing + preferred devices), `SetInputDevice`/`SetOutputDevice`. iOS/Mac Catalyst + Android + Linux only - Audio route enumeration/selection via `IAudioDevices` — `GetInputs`/`GetOutputs`, `CurrentInput`/`CurrentOutput`, `Changed` event; normalized `AudioDevice.Type`. iOS/Mac Catalyst + Android + Linux only (`Changed` needs PulseAudio/PipeWire; `ShowOutputPicker()` is a no-op on Linux) - Linux support via the separate `Shiny.Audio.Linux` package — all four audio services over PulseAudio/PipeWire with an ALSA fallback. **No native STT/TTS exists on Linux** (there is no OS speech engine to wrap); use a cloud provider, or `Shiny.Speech.Linux.Whisper` for offline STT - On-device offline STT on Linux via the separate `Shiny.Speech.Linux.Whisper` package — Whisper through whisper.cpp, no cloud account or network. STT only - Route classification via `AudioDeviceExtensions` — `IsWired()`, `IsBluetooth()`, `IsBuiltIn()`, `IsHeadphones()`, `HasMicrophone()` on both `AudioDevice` and `AudioDeviceType`; detects wired/jack/USB-C headphones and headsets alongside Bluetooth - One-stop `IAudio` facade exposing `Player` / `Source` / `Monitor` / `Devices` — inject it to discover the whole audio surface (focused interfaces remain independently injectable) - Pluggable cloud provider architecture via `ISpeechToTextProvider` and `ITextToSpeechProvider` - Azure AI Speech integration (STT + TTS) - ElevenLabs integration (Scribe STT + TTS) - Typecast integration (TTS only) - Convenience extension methods: `ListenUntilSilence`, `StatementAfterKeyword`, `WaitListenForKeywords`, `ListenForKeywords` - Permission management via `AccessState` and `RequestAccess()` - VU meter signal (outgoing) — `AudioLevelChanged` event on `ITextToSpeechService` and `IAudioPlayer` emits a normalized 0.0–1.0 RMS level during playback; `IsPlayerAnalysisSupported` reports per-platform availability - VU meter signal (incoming) — `InputLevelChanged` event on `ISpeechToTextService` (gated by `IsInputAnalysisSupported`), `IAudioSource`, and `IAudioMonitor` emits the same normalized 0.0–1.0 mic level while listening / capturing / monitoring - `AudioLevel` helper — the shared dBFS mapping (`FromRms` / `FromPcm16` / `FromSamples`) used by every meter, public so PCM you consume yourself meters on the same scale ## Setup ### 1. Install NuGet Packages For platform-native speech only: ```bash dotnet add package Shiny.Speech ``` For Azure AI Speech (cloud STT + TTS): ```bash dotnet add package Shiny.Speech dotnet add package Shiny.Speech.Azure ``` For ElevenLabs (cloud TTS): ```bash dotnet add package Shiny.Speech dotnet add package Shiny.Speech.ElevenLabs ``` For Linux (console app, daemon, container, Raspberry Pi) — add the Linux audio backend alongside a recognizer, because Linux has no native speech engine: ```bash dotnet add package Shiny.Audio.Linux dotnet add package Shiny.Speech.Azure # or .OpenAI / .ElevenLabs / .Typecast # ...or run recognition offline on the device itself (STT only): dotnet add package Shiny.Speech.Linux.Whisper ``` ### 2. Configure in MauiProgram.cs (or Blazor Program.cs) **Platform-native speech services:** ```csharp builder .UseMauiApp<App>() .UseShiny(); // REQUIRED for native STT/audio — registers Shiny.Core's AndroidPlatform builder.Services.AddSpeechServices(); // Registers STT, TTS, AudioSource, AudioPlayer // On Browser/WASM: auto-detected via OperatingSystem.IsBrowser() ``` > **`UseShiny()` is mandatory for native speech/audio.** Android runtime permission requests > (`RECORD_AUDIO`) and current-activity tracking are delegated to `Shiny.Core`'s `AndroidPlatform`. > Reference the `Shiny.Hosting.Maui` package and call `.UseShiny()` on the `MauiAppBuilder` (before > building) so `AndroidPlatform` is registered and receives `OnRequestPermissionsResult` callbacks. > Without it, `RequestAccess()` throws a `TimeoutException` ("no current activity"). This replaced the > library's old self-contained `ActivityProvider` + `PermissionRequestFragment`. Or register individually: ```csharp builder.Services.AddSpeechToText(); // ISpeechToTextService only builder.Services.AddTextToSpeech(); // ITextToSpeechService only builder.Services.AddAudioServices(); // IAudioSource + IAudioPlayer + IAudioMonitor + IAudioDevices + IAudio (from Shiny.Audio) builder.Services.AddAudioSource(); // IAudioSource only builder.Services.AddAudioPlayer(); // IAudioPlayer only builder.Services.AddAudioMonitor(); // IAudioMonitor only (iOS/Mac Catalyst + Android) builder.Services.AddAudioDevices(); // IAudioDevices only (iOS/Mac Catalyst + Android) ``` > **Namespace:** `IAudioSource`, `IAudioPlayer`, and `PipeStream` live in the **`Shiny.Audio`** > namespace (shipped in the standalone `Shiny.Audio` package, referenced by `Shiny.Speech`). Add > `using Shiny.Audio;` when consuming them. `AccessState` now comes from **`Shiny.Core`** and lives in > the **`Shiny`** namespace — because `Shiny` is a parent namespace, code inside `Shiny.Audio`/ > `Shiny.Speech` resolves it automatically; add `using Shiny;` only where you reference it elsewhere. > All the DI extension methods above are in the `Shiny` namespace regardless of package. > `AddAudioServices()` / `AddAudioSource()` / `AddAudioPlayer()` come from `Shiny.Audio` and can be used > **without** `Shiny.Speech` for capture/playback-only scenarios (still require `.UseShiny()` on > Android). **Azure AI Speech (replaces platform-native with cloud):** ```csharp builder.Services.AddAzureSpeech("your-subscription-key", "eastus"); // Automatically registers IAudioSource and IAudioPlayer for platform audio I/O ``` Or with config object and selective services: ```csharp builder.Services.AddAzureSpeech( new AzureSpeechConfig { SubscriptionKey = "key", Region = "eastus" }, speechToText: true, textToSpeech: true ); ``` **ElevenLabs (replaces platform-native STT/TTS with cloud — Scribe + TTS):** ```csharp // Register both STT (Scribe) and TTS at once builder.Services.AddElevenLabsSpeech("your-api-key"); // Or selectively builder.Services.AddElevenLabsSpeechToText("your-api-key"); // Scribe STT only builder.Services.AddElevenLabsTextToSpeech("your-api-key"); // TTS only // Auto-registers IAudioSource and/or IAudioPlayer for platform audio I/O as needed ``` ```csharp // With a config object — overrides default Scribe model / TTS model / voice builder.Services.AddElevenLabsSpeech(new ElevenLabsConfig { ApiKey = "your-api-key", SpeechToTextModel = "scribe_v1", TextToSpeechModel = "eleven_multilingual_v2", DefaultVoiceId = "21m00Tcm4TlvDq8ikWAM" }); ``` > **ElevenLabs Scribe is request/response, not streaming**: results are yielded as a single final `SpeechRecognitionResult` when the user calls `Stop()` (the captured audio is buffered, wrapped in a WAV container, and posted to `/v1/speech-to-text`). For continuous partial results, use Azure instead. **Typecast (cloud TTS only — via the `typecast-csharp` SDK):** ```csharp builder.Services.AddTypecastSpeech("your-typecast-api-key"); // AddTypecastTextToSpeech(...) is an identical alias. Registers ITextToSpeechService + IAudioPlayer. // With a config object — model, default voice, language, emotion, audio format: builder.Services.AddTypecastSpeech(new TypecastConfig { ApiKey = "your-typecast-api-key", DefaultVoiceId = "<voice-id>", // required unless you pass TextToSpeechOptions.Voice per call Model = Typecast.Models.TTSModel.SsfmV30, AudioFormat = Typecast.Models.AudioFormat.Mp3 }); ``` > **Typecast is TTS-only** — there is no `AddTypecastSpeechToText`; pair it with Azure/ElevenLabs/OpenAI or native STT if you need recognition. It has **no fixed default voice**: set `TypecastConfig.DefaultVoiceId` or pass `TextToSpeechOptions.Voice`, and call `ITextToSpeechService.GetVoicesAsync()` to discover the voice ids available to your account. `TypecastConfig` also exposes optional `Language`, `Emotion` (+`EmotionIntensity`) hints. **Changing API keys / credentials at runtime:** All cloud provider config objects (`AzureSpeechConfig`, `ElevenLabsConfig`, `OpenAiSpeechConfig`, `TypecastConfig`) are **mutable singletons**. Register them normally, then change the key (or region/model/voice) at any time — the provider uses the new value on its next call, no re-registration needed. Keep a reference to the config you pass in, or resolve it from DI: ```csharp var config = new TypecastConfig { ApiKey = "initial" }; builder.Services.AddTypecastSpeech(config); // ...later: config.ApiKey = "rotated-key"; // via your retained reference serviceProvider.GetRequiredService<AzureSpeechConfig>().SubscriptionKey = "new-key"; // or resolve from DI ``` Providers that cache an SDK/HTTP client (ElevenLabs, Typecast) rebuild it automatically when the key changes (via `RefreshableClient<T>` in `Shiny.Speech.Cloud`); Azure and OpenAI read the config on every call. Do **not** re-call `AddXxxSpeech(...)` to change a key — just mutate the config. ### 3. Platform Permissions **Android** — Add to `AndroidManifest.xml`: ```xml <uses-permission android:name="android.permission.RECORD_AUDIO" /> <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" /> ``` `MODIFY_AUDIO_SETTINGS` is required for the TTS audio-level Visualizer and for the native STT beep suppression. Do **not** add a `<queries>` element for `android.speech.RecognitionService` — `Shiny.Speech` ships its own library manifest with it and that merges into the app automatically. (Without that declaration, API 30+ package visibility hides the recognition service, `IsSupported` is `false` and `RequestAccess()` returns `AccessState.NotSupported` before the mic permission is ever requested. The library handles it; consumers should not duplicate it.) **iOS** — Add to `Info.plist`: ```xml <key>NSSpeechRecognitionUsageDescription</key> <string>This app uses speech recognition</string> <key>NSMicrophoneUsageDescription</key> <string>This app uses the microphone for speech recognition</string> ``` **Browser (Blazor WebAssembly)** — No manifest changes and **no `<script>` tag** needed. The browser prompts for microphone access automatically, and the JS interop module ships **inside the `Shiny.Audio` package** as a static web asset (`_content/Shiny.Audio/shiny-audio.js`), loaded on demand via `JSHost.ImportAsync`. Do **not** copy the JS into `wwwroot` or add a `<script>` reference — just reference the NuGet package. > **Note:** `IAudioSource` captures raw PCM audio in the browser using the Web Audio API (`getUserMedia` + `ScriptProcessorNode`), downsampled to 16kHz 16-bit mono — the same format as other platforms. **Linux** — No manifest or runtime permission prompt; microphone access is a filesystem/group concern the distro handles. Audio services come from the separate `Shiny.Audio.Linux` package and must be registered explicitly: ```csharp using Shiny; // MUST come before AddSpeechServices() / AddCloudSpeechToText<T>() / AddCloudTextToSpeech<T>(). // Those use TryAdd, so the first registration wins — and on Linux they register nothing at all. // AddLinuxAudio() is a no-op off Linux, so it is safe in shared startup code. builder.Services.AddLinuxAudio(); builder.Services.AddCloudSpeechToText<AzureSpeechToTextProvider>(); builder.Services.AddCloudTextToSpeech<AzureTextToSpeechProvider>(); ``` **Linux offline STT (Whisper)** — when the user wants recognition without a cloud account, an API key, or a network connection (air-gapped boxes, kiosks, Raspberry Pi appliances, privacy requirements), use `Shiny.Speech.Linux.Whisper` instead of a cloud provider: ```csharp using Shiny; using Shiny.Speech.Linux; using Whisper.net.Ggml; builder.Services.AddLinuxAudio(); // MUST come first — supplies IAudioSource // Simple: pick a model size. builder.Services.AddLinuxWhisperSpeechToText(GgmlType.BaseEn, QuantizationType.Q5_1); // Or configure fully: builder.Services.AddLinuxWhisperSpeechToText(new WhisperConfig { ModelType = GgmlType.BaseEn, Quantization = QuantizationType.Q5_1, InitialPrompt = "Shiny, MAUI, Blazor", // bias the decoder toward domain vocabulary SilenceRmsThreshold = 500 // VAD sensitivity (0–32767) }); ``` It registers a normal `ISpeechToTextService`, so `ListenUntilSilence()`, `StatementAfterKeyword()`, `WaitListenForKeywords()` and the `KeywordHeard` event all work unchanged. When generating Whisper code: - **STT only.** Whisper is a recognition model — there is no Whisper TTS. If the user also wants speech output, pair it with a cloud TTS provider (`AddCloudTextToSpeech<T>()`). Never generate an `AddLinuxWhisperTextToSpeech` — it does not exist. - **Never generate partial-result handling.** Whisper is a batch model; every `SpeechRecognitionResult` has `IsFinal = true`. The provider runs client-side VAD and turns each speech→silence segment into one inference. - **Recommend `PrepareAsync()` at startup.** The model downloads from Hugging Face on first use