shiny-speech

name	shiny-speech
description	Generate code using Shiny.Speech for cross-platform speech-to-text, text-to-speech, audio capture, and audio playback with pluggable cloud providers
auto_invoke	true
triggers	["speech to text","text to speech","speech recognition","voice recognition","tts","stt","speak","dictation","transcribe","synthesize speech","audio capture","audio playback","microphone","ISpeechToTextService","ITextToSpeechService","SpeechTone","SpeechEmotion","SpeechAnnotations","SpeechAnnotationHandling","SpeechToneCapabilities","audio tags","emotion","emotional tts","expressive speech","speech tone","eleven v3","emotion_preset","express-as","IAudioSource","IAudioPlayer","IAudioPlayback","concurrent playback","[Truncated]"]

Shiny Speech Skill

You are an expert in Shiny Speech, a library that provides cross-platform speech-to-text, text-to-speech, audio capture, and audio playback for .NET MAUI and Blazor WebAssembly with pluggable cloud providers.

When to Use This Skill

Invoke this skill when the user wants to:

• Add speech-to-text (STT) or text-to-speech (TTS) to a .NET MAUI app
• Capture audio from the device microphone
• Play audio streams on the device
• Use Azure AI Speech for cloud-based STT/TTS
• Use ElevenLabs for cloud-based TTS
• Implement a custom cloud speech provider
• Configure speech recognition options (language, silence timeout, on-device preference, keywords)
• Configure text-to-speech options (voice, rate, pitch, volume)
• Add emotion / tone / expressive delivery to synthesized speech
• Use ElevenLabs v3 audio tags ([excited], [whispers]) or Typecast emotion presets
• Strip or normalize bracketed audio tags so they aren't spoken aloud
• List available TTS voices
• Start/stop continuous speech recognition with event-based results
• Implement listen-until-silence dictation
• Implement wake word / keyword activation ("Hey Siri" style)
• Implement keyword listening (listen until a specific keyword is detected)
• Listen for keywords continuously as an async stream
• Add speech-to-text or text-to-speech to a Blazor WebAssembly app
• Use the Web Speech API via Shiny.Speech in the browser

Library Overview

GitHub: https://github.com/shinyorg/speech NuGet Packages:

• Shiny.Speech — Core library with platform-native STT, TTS, audio capture, and playback
• Shiny.Speech.Cloud — Cloud provider abstractions
• Shiny.Speech.Azure — Azure AI Speech provider
• Shiny.Speech.ElevenLabs — ElevenLabs TTS provider
• Shiny.Speech.Typecast — Typecast TTS provider (TTS only, via the typecast-csharp SDK)
• Shiny.Audio.Linux — Linux audio backend (PulseAudio/PipeWire with ALSA fallback). Required on Linux; nothing else registers audio services there
• Shiny.Speech.Linux.Whisper — On-device, offline STT on Linux via Whisper/whisper.cpp. STT only (there is no Whisper TTS)

Namespace: Shiny.Speech

Shiny Speech provides:

• Platform-native speech-to-text via ISpeechToTextService (iOS, Android, Windows, Browser/WASM)
• Event-based recognition model — ResultReceived, KeywordHeard, Error events allow multiple subscribers
• Start/Stop lifecycle — call Start() to begin listening, Stop() to end; Start() throws if already listening
• Built-in keyword detection — set Keywords in SpeechRecognitionOptions and subscribe to KeywordHeard
• Platform-native text-to-speech via ITextToSpeechService (iOS, Android, Windows, Browser/WASM)
• Platform-native audio capture via IAudioSource (raw PCM 16kHz, 16-bit, mono — all platforms including browser) with an InputLevelChanged VU signal on every platform
• Platform-native audio playback via IAudioPlayer — play a Stream, or a remote URL / local file path via PlayAsync(string) (platform resolves the source natively; browser uses HTML5 Audio)
• Concurrent playback — clips overlap instead of interrupting each other; StartAsync returns an IAudioPlayback handle (Id, Source, IsPlaying, Completion, StopAsync()) that stops one clip on its own, IAudioPlayer.Active lists what is playing, and IAudioPlayer.StopAsync() stops everything
• Live microphone monitor via IAudioMonitor — routes the mic to the current output in near-real-time (PA / "talk over a Bluetooth speaker"); Start/Stop, adjustable Gain, InputLevelChanged VU signal, AudioMonitorOptions (voice processing + preferred devices), SetInputDevice/SetOutputDevice. iOS/Mac Catalyst + Android + Linux only
• Audio route enumeration/selection via IAudioDevices — GetInputs/GetOutputs, CurrentInput/CurrentOutput, Changed event; normalized AudioDevice.Type. iOS/Mac Catalyst + Android + Linux only (Changed needs PulseAudio/PipeWire; ShowOutputPicker() is a no-op on Linux)
• Linux support via the separate Shiny.Audio.Linux package — all four audio services over PulseAudio/PipeWire with an ALSA fallback. No native STT/TTS exists on Linux (there is no OS speech engine to wrap); use a cloud provider, or Shiny.Speech.Linux.Whisper for offline STT
• On-device offline STT on Linux via the separate Shiny.Speech.Linux.Whisper package — Whisper through whisper.cpp, no cloud account or network. STT only
• Route classification via AudioDeviceExtensions — IsWired(), IsBluetooth(), IsBuiltIn(), IsHeadphones(), HasMicrophone() on both AudioDevice and AudioDeviceType; detects wired/jack/USB-C headphones and headsets alongside Bluetooth
• One-stop IAudio facade exposing Player / Source / Monitor / Devices — inject it to discover the whole audio surface (focused interfaces remain independently injectable)
• Pluggable cloud provider architecture via ISpeechToTextProvider and ITextToSpeechProvider
• Azure AI Speech integration (STT + TTS)
• ElevenLabs integration (Scribe STT + TTS)
• Typecast integration (TTS only)
• Convenience extension methods: ListenUntilSilence, StatementAfterKeyword, WaitListenForKeywords, ListenForKeywords
• Permission management via AccessState and RequestAccess()
• VU meter signal (outgoing) — AudioLevelChanged event on ITextToSpeechService and IAudioPlayer emits a normalized 0.0–1.0 RMS level during playback; IsPlayerAnalysisSupported reports per-platform availability
• VU meter signal (incoming) — InputLevelChanged event on ISpeechToTextService (gated by IsInputAnalysisSupported), IAudioSource, and IAudioMonitor emits the same normalized 0.0–1.0 mic level while listening / capturing / monitoring
• AudioLevel helper — the shared dBFS mapping (FromRms / FromPcm16 / FromSamples) used by every meter, public so PCM you consume yourself meters on the same scale

Setup

1. Install NuGet Packages

For platform-native speech only:

dotnet add package Shiny.Speech

For Azure AI Speech (cloud STT + TTS):

dotnet add package Shiny.Speech
dotnet add package Shiny.Speech.Azure

For ElevenLabs (cloud TTS):

dotnet add package Shiny.Speech
dotnet add package Shiny.Speech.ElevenLabs

For Linux (console app, daemon, container, Raspberry Pi) — add the Linux audio backend alongside a recognizer, because Linux has no native speech engine:

dotnet add package Shiny.Audio.Linux
dotnet add package Shiny.Speech.Azure          # or .OpenAI / .ElevenLabs / .Typecast
# ...or run recognition offline on the device itself (STT only):
dotnet add package Shiny.Speech.Linux.Whisper

2. Configure in MauiProgram.cs (or Blazor Program.cs)

Platform-native speech services:

builder
    .UseMauiApp<App>()
    .UseShiny(); // REQUIRED for native STT/audio — registers Shiny.Core's AndroidPlatform

builder.Services.AddSpeechServices(); // Registers STT, TTS, AudioSource, AudioPlayer
// On Browser/WASM: auto-detected via OperatingSystem.IsBrowser()

UseShiny() is mandatory for native speech/audio. Android runtime permission requests (RECORD_AUDIO) and current-activity tracking are delegated to Shiny.Core's AndroidPlatform. Reference the Shiny.Hosting.Maui package and call .UseShiny() on the MauiAppBuilder (before building) so AndroidPlatform is registered and receives OnRequestPermissionsResult callbacks. Without it, RequestAccess() throws a TimeoutException ("no current activity"). This replaced the library's old self-contained ActivityProvider + PermissionRequestFragment.

Or register individually:

builder.Services.AddSpeechToText();   // ISpeechToTextService only
builder.Services.AddTextToSpeech();   // ITextToSpeechService only
builder.Services.AddAudioServices();  // IAudioSource + IAudioPlayer + IAudioMonitor + IAudioDevices + IAudio (from Shiny.Audio)
builder.Services.AddAudioSource();    // IAudioSource only
builder.Services.AddAudioPlayer();    // IAudioPlayer only
builder.Services.AddAudioMonitor();   // IAudioMonitor only (iOS/Mac Catalyst + Android)
builder.Services.AddAudioDevices();   // IAudioDevices only (iOS/Mac Catalyst + Android)

Namespace: IAudioSource, IAudioPlayer, and PipeStream live in the Shiny.Audio namespace (shipped in the standalone Shiny.Audio package, referenced by Shiny.Speech). Add using Shiny.Audio; when consuming them. AccessState now comes from Shiny.Core and lives in the Shiny namespace — because Shiny is a parent namespace, code inside Shiny.Audio/ Shiny.Speech resolves it automatically; add using Shiny; only where you reference it elsewhere. All the DI extension methods above are in the Shiny namespace regardless of package. AddAudioServices() / AddAudioSource() / AddAudioPlayer() come from Shiny.Audio and can be used without Shiny.Speech for capture/playback-only scenarios (still require .UseShiny() on Android).

Azure AI Speech (replaces platform-native with cloud):

builder.Services.AddAzureSpeech("your-subscription-key", "eastus");
// Automatically registers IAudioSource and IAudioPlayer for platform audio I/O

Or with config object and selective services:

builder.Services.AddAzureSpeech(
    new AzureSpeechConfig { SubscriptionKey = "key", Region = "eastus" },
    speechToText: true,
    textToSpeech: true
);

ElevenLabs (replaces platform-native STT/TTS with cloud — Scribe + TTS):

// Register both STT (Scribe) and TTS at once
builder.Services.AddElevenLabsSpeech("your-api-key");

// Or selectively
builder.Services.AddElevenLabsSpeechToText("your-api-key"); // Scribe STT only
builder.Services.AddElevenLabsTextToSpeech("your-api-key"); // TTS only
// Auto-registers IAudioSource and/or IAudioPlayer for platform audio I/O as needed

// With a config object — overrides default Scribe model / TTS model / voice
builder.Services.AddElevenLabsSpeech(new ElevenLabsConfig
{
    ApiKey = "your-api-key",
    SpeechToTextModel = "scribe_v1",
    TextToSpeechModel = "eleven_multilingual_v2",
    DefaultVoiceId = "21m00Tcm4TlvDq8ikWAM"
});

ElevenLabs Scribe is request/response, not streaming: results are yielded as a single final SpeechRecognitionResult when the user calls Stop() (the captured audio is buffered, wrapped in a WAV container, and posted to /v1/speech-to-text). For continuous partial results, use Azure instead.

Typecast (cloud TTS only — via the typecast-csharp SDK):

builder.Services.AddTypecastSpeech("your-typecast-api-key");
// AddTypecastTextToSpeech(...) is an identical alias. Registers ITextToSpeechService + IAudioPlayer.

// With a config object — model, default voice, language, emotion, audio format:
builder.Services.AddTypecastSpeech(new TypecastConfig
{
    ApiKey = "your-typecast-api-key",
    DefaultVoiceId = "<voice-id>",           // required unless you pass TextToSpeechOptions.Voice per call
    Model = Typecast.Models.TTSModel.SsfmV30,
    AudioFormat = Typecast.Models.AudioFormat.Mp3
});

Typecast is TTS-only — there is no AddTypecastSpeechToText; pair it with Azure/ElevenLabs/OpenAI or native STT if you need recognition. It has no fixed default voice: set TypecastConfig.DefaultVoiceId or pass TextToSpeechOptions.Voice, and call ITextToSpeechService.GetVoicesAsync() to discover the voice ids available to your account. TypecastConfig also exposes optional Language, Emotion (+EmotionIntensity) hints.

Changing API keys / credentials at runtime: All cloud provider config objects (AzureSpeechConfig, ElevenLabsConfig, OpenAiSpeechConfig, TypecastConfig) are mutable singletons. Register them normally, then change the key (or region/model/voice) at any time — the provider uses the new value on its next call, no re-registration needed. Keep a reference to the config you pass in, or resolve it from DI:

var config = new TypecastConfig { ApiKey = "initial" };
builder.Services.AddTypecastSpeech(config);
// ...later:
config.ApiKey = "rotated-key";                              // via your retained reference
serviceProvider.GetRequiredService<AzureSpeechConfig>().SubscriptionKey = "new-key"; // or resolve from DI

Providers that cache an SDK/HTTP client (ElevenLabs, Typecast) rebuild it automatically when the key changes (via RefreshableClient<T> in Shiny.Speech.Cloud); Azure and OpenAI read the config on every call. Do not re-call AddXxxSpeech(...) to change a key — just mutate the config.

3. Platform Permissions

Android — Add to AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

MODIFY_AUDIO_SETTINGS is required for the TTS audio-level Visualizer and for the native STT beep suppression.

Do not add a <queries> element for android.speech.RecognitionService — Shiny.Speech ships its own library manifest with it and that merges into the app automatically. (Without that declaration, API 30+ package visibility hides the recognition service, IsSupported is false and RequestAccess() returns AccessState.NotSupported before the mic permission is ever requested. The library handles it; consumers should not duplicate it.)

iOS — Add to Info.plist:

<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses speech recognition</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app uses the microphone for speech recognition</string>

Browser (Blazor WebAssembly) — No manifest changes and no <script> tag needed. The browser prompts for microphone access automatically, and the JS interop module ships inside the Shiny.Audio package as a static web asset (_content/Shiny.Audio/shiny-audio.js), loaded on demand via JSHost.ImportAsync. Do not copy the JS into wwwroot or add a <script> reference — just reference the NuGet package.

Note: IAudioSource captures raw PCM audio in the browser using the Web Audio API (getUserMedia + ScriptProcessorNode), downsampled to 16kHz 16-bit mono — the same format as other platforms.

Linux — No manifest or runtime permission prompt; microphone access is a filesystem/group concern the distro handles. Audio services come from the separate Shiny.Audio.Linux package and must be registered explicitly:

using Shiny;

// MUST come before AddSpeechServices() / AddCloudSpeechToText<T>() / AddCloudTextToSpeech<T>().
// Those use TryAdd, so the first registration wins — and on Linux they register nothing at all.
// AddLinuxAudio() is a no-op off Linux, so it is safe in shared startup code.
builder.Services.AddLinuxAudio();

builder.Services.AddCloudSpeechToText<AzureSpeechToTextProvider>();
builder.Services.AddCloudTextToSpeech<AzureTextToSpeechProvider>();

Linux offline STT (Whisper) — when the user wants recognition without a cloud account, an API key, or a network connection (air-gapped boxes, kiosks, Raspberry Pi appliances, privacy requirements), use Shiny.Speech.Linux.Whisper instead of a cloud provider:

using Shiny;
using Shiny.Speech.Linux;
using Whisper.net.Ggml;

builder.Services.AddLinuxAudio();   // MUST come first — supplies IAudioSource

// Simple: pick a model size.
builder.Services.AddLinuxWhisperSpeechToText(GgmlType.BaseEn, QuantizationType.Q5_1);

// Or configure fully:
builder.Services.AddLinuxWhisperSpeechToText(new WhisperConfig
{
    ModelType = GgmlType.BaseEn,
    Quantization = QuantizationType.Q5_1,
    InitialPrompt = "Shiny, MAUI, Blazor",   // bias the decoder toward domain vocabulary
    SilenceRmsThreshold = 500                // VAD sensitivity (0–32767)
});

It registers a normal ISpeechToTextService, so ListenUntilSilence(), StatementAfterKeyword(), WaitListenForKeywords() and the KeywordHeard event all work unchanged.

When generating Whisper code:

• STT only. Whisper is a recognition model — there is no Whisper TTS. If the user also wants speech output, pair it with a cloud TTS provider (AddCloudTextToSpeech<T>()). Never generate an AddLinuxWhisperTextToSpeech — it does not exist.
• Never generate partial-result handling. Whisper is a batch model; every SpeechRecognitionResult has IsFinal = true. The provider runs client-side VAD and turns each speech→silence segment into one inference.
• Recommend PrepareAsync() at startup. The model downloads from Hugging Face on first use