# Shiny.Speech & Shiny.AiConversation
This repository is the home for two complementary library families:
- **Shiny.Speech** โ Cross-platform speech services for .NET MAUI and Blazor WebAssembly: speech-to-text and text-to-speech with pluggable cloud providers. Audio capture and playback are provided by the standalone **Shiny.Audio** package (referenced automatically).
- **Shiny.Audio** โ Cross-platform microphone capture (`IAudioSource`) and concurrent stream playback (`IAudioPlayer` / `IAudioPlayback`) with VU-level metering, a live mic-to-output **monitor** (`IAudioMonitor`), audio **route enumeration/selection** (`IAudioDevices`), WAV **recording** (`IAudioRecorder`), real-time capture **effects** (`AudioEffectChain` โ pitch, echo, reverb, filters), and echo-cancelled **full-duplex voice** for calls (`IAudioDuplex`), all discoverable through one `IAudio` facade. Usable on its own; also the audio backbone for Shiny.Speech.
- **Shiny.AiConversation** โ Session-based AI conversations over Microsoft.Extensions.AI: streamed replies, typed questions, persisted and resumable sessions, compaction, assistant memory and token/context tracking, with voice (wake word, push-to-talk, interruption, text-to-speech), a MAUI chat UI and a camera tool layered on top. AiConversation drives much of the real-world feature set (and bug surface) of the speech stack, which is why both live and ship from here together.
All packages share a single version, defined by `version.json` at the repo root (Nerdbank.GitVersioning).
๐ **[Try the live Blazor WebAssembly playground](https://shinyorg.github.io/speech/)** โ speech-to-text, text-to-speech, and
the AI conversation loop running in the browser. Source: [`samples/BlazorSample`](samples/BlazorSample).
๐ **Documentation:** [Speech](https://shinylib.net/speech/) ยท [Audio](https://shinylib.net/audio/) ยท [AI Conversations](https://shinylib.net/aiconversation/)
## Libraries
| Package | Description | Targets |
|---------|-------------|---------|
| **Shiny.Audio** | Standalone audio capture (`IAudioSource`) + playback (`IAudioPlayer`) + live mic monitor (`IAudioMonitor`) + route enumeration (`IAudioDevices`) + WAV recording (`IAudioRecorder`) + real-time capture effects (`AudioEffectChain`) + full-duplex call audio (`IAudioDuplex`) behind the `IAudio` facade, with native platform implementations | net10.0-ios, net10.0-android, net10.0-windows, net10.0 (Browser/WASM) |
| **Shiny.Audio.Linux** | Linux backend for all four audio services over PulseAudio/PipeWire with an ALSA fallback. Ships separately because it carries a managed MP3 decoder (`NLayer`) โ Linux has no system decoder to call | net10.0 |
| **Shiny.Speech** | Core STT/TTS interfaces + native platform implementations (references Shiny.Audio for capture/playback) | net10.0-ios, net10.0-android, net10.0-windows, net10.0 (Browser/WASM) |
| **Shiny.Speech.Cloud** | Cloud provider abstractions + `CloudSpeechToText` / `CloudTextToSpeech` implementations | net10.0 |
| **Shiny.Speech.Azure** | Azure AI Speech provider (STT + TTS) | net10.0 |
| **Shiny.Speech.ElevenLabs** | ElevenLabs provider (STT + TTS) | net10.0 |
| **Shiny.Speech.Typecast** | Typecast provider (TTS only, via the `typecast-csharp` SDK) | net10.0 |
| **Shiny.Speech.Linux.Whisper** | On-device, offline STT for Linux (incl. Raspberry Pi) running Whisper locally through whisper.cpp โ no cloud account, no network | net10.0 |
| **Shiny.AiConversation** | `IAiSessionManager` / `IAiSession`: persisted, streaming AI sessions with typed questions and context tracking (no audio dependency) | net10.0 |
| **Shiny.AiConversation.Voice** | `IVoiceConversation`: push-to-talk, wake word, follow-up, interruption and sentence-streamed text-to-speech over Shiny.Speech | net10.0 |
| **Shiny.AiConversation.Maui** | `AiChatView` (a Shiny.Maui.Controls `ChatView` bound to a session) and `AiMemoryView` | net10.0 (MAUI) |
| **Shiny.AiConversation.Maui.Camera** | The `use_camera` tool over Shiny's CameraView modal (`IMediaService`) | net10.0 (MAUI) |
| **Shiny.AiConversation.OpenAi** | Ready-made static OpenAI-compatible chat client provider | net10.0 |
| **Shiny.AiConversation.Maui.GithubCopilot** | MAUI GitHub Copilot provider (device-code OAuth, SecureStorage) | net10.0 (MAUI) |
| **Shiny.AiConversation.Stores.SqliteDocDb** | SQLite (Shiny.DocumentDb) session, message and memory stores with full-text search (renamed from `MessageStores.SqliteDocDb`) | net10.0 |
## Getting Started
### Native Platform Speech
Use the built-in OS speech engines โ no cloud account needed. Works on MAUI (iOS, Android, Windows) and Blazor WebAssembly (via Web Speech API).
```csharp
builder
.UseMauiApp<App>()
.UseShiny(); // required โ see note below
builder.Services.AddSpeechServices();
// Registers: ISpeechToTextService, ITextToSpeechService, IAudioSource, IAudioPlayer
// On Browser/WASM: auto-detected via OperatingSystem.IsBrowser()
```
> **`UseShiny()` is now required for native speech/audio.** Runtime permission handling (Android
> `RECORD_AUDIO`) and activity tracking are delegated to [Shiny.Core](https://shinylib.net)'s
> `AndroidPlatform` instead of a hand-rolled fragment. Add the `Shiny.Hosting.Maui` package and call
> `.UseShiny()` on the `MauiAppBuilder` so `AndroidPlatform` is registered and receives permission
> callbacks. This replaces the old self-contained `ActivityProvider`/`PermissionRequestFragment`.
> **`AccessState` now comes from `Shiny.Core`.** It lives in the `Shiny` namespace (it previously lived
> in `Shiny.Audio`). Because `Shiny` is a parent of `Shiny.Audio`/`Shiny.Speech`, most code needs no
> change; add `using Shiny;` only where you reference it outside those namespaces. `IAudioSource`,
> `IAudioPlayer`, and `PipeStream` remain in `Shiny.Audio` โ add `using Shiny.Audio;` where you consume
> them. `AddSpeechServices()` still wires up capture + playback; to register audio on its own โ including
> the live monitor (`IAudioMonitor`), route enumeration (`IAudioDevices`), recording (`IAudioRecorder`),
> and the `IAudio` facade โ call `builder.Services.AddAudioServices();`.
### Azure AI Speech (Cloud)
```csharp
builder.Services.AddAzureSpeech("your-subscription-key", "your-region");
```
### ElevenLabs (Cloud)
```csharp
// Register both STT (Scribe) and TTS:
builder.Services.AddElevenLabsSpeech("your-api-key");
// Or pick one:
builder.Services.AddElevenLabsSpeechToText("your-api-key");
builder.Services.AddElevenLabsTextToSpeech("your-api-key");
```
### Typecast (Cloud, TTS only)
```csharp
builder.Services.AddTypecastSpeech("your-typecast-api-key");
// Or configure the model / default voice / audio format:
builder.Services.AddTypecastSpeech(new TypecastConfig
{
ApiKey = "your-typecast-api-key",
DefaultVoiceId = "<voice-id>", // call ITextToSpeechService.GetVoicesAsync() to discover ids
Model = TTSModel.SsfmV30
});
```
> Typecast has no fixed public default voice โ set `DefaultVoiceId` (or pass a `VoiceInfo` per call via
> `TextToSpeechOptions.Voice`). Use `GetVoicesAsync()` to list the voice ids available to your account.
### Changing credentials at runtime
You configure providers exactly as above โ but the config objects are **mutable singletons**, so you can
change the API key (or region / model / voice) at any time and the provider picks it up on its next call.
No re-registration required. Hold your config instance, or resolve it from DI:
```csharp
var config = new AzureSpeechConfig { SubscriptionKey = "initial-key", Region = "eastus" };
builder.Services.AddAzureSpeech(config);
// ...later, e.g. after the user pastes a new key in settings:
config.SubscriptionKey = "rotated-key"; // next Speak/recognize uses it
// Or resolve it from the container if you didn't keep a reference:
serviceProvider.GetRequiredService<TypecastConfig>().ApiKey = "new-key";
```
Providers that cache an SDK/HTTP client (ElevenLabs, Typecast) transparently rebuild it when the key
changes; Azure and OpenAI read the config on every call.
## Usage
### Text-to-Speech
```csharp
public class MyService(ITextToSpeechService tts)
{
public async Task SpeakAsync()
{
await tts.SpeakAsync("Hello world!", new TextToSpeechOptions
{
SpeechRate = 1.2f,
Pitch = 1.0f,
Volume = 0.8f
});
}
}
```
### Emotion & Tone
Expressive TTS engines expose emotion in incompatible ways โ ElevenLabs v3 reads inline audio tags
(`[excited]`) out of the text, Typecast takes an `emotion_preset` field, Azure uses SSML
`mstts:express-as`, and OpenAI takes a free-text `instructions` string. `SpeechTone` is the portable
form; every provider projects it onto whatever it actually supports, and providers with no
expressive control ignore it:
```csharp
await tts.SpeakAsync("We just shipped it.", new TextToSpeechOptions
{
Tone = new SpeechTone
{
Emotion = SpeechEmotion.Excited,
Intensity = 1.5f, // Typecast emotion_intensity / Azure styledegree
Instructions = "Sound like you're sharing good news." // OpenAI instructions
}
});
```
Bracketed annotations in the text are handled too. Because only ElevenLabs' `eleven_v3` performs
them โ every other engine, including older ElevenLabs models, reads them aloud verbatim โ the
default `SpeechAnnotationHandling.Auto` promotes the first emotion tag to a `SpeechTone` and then
strips the tags for any provider that can't perform them:
```csharp
// Speaks the tags as direction on eleven_v3; everywhere else says "We just shipped it. Everything is live."
// with the closest available emotion applied.
await tts.SpeakAsync("[excited] We just shipped it. [laughs] Everything is live.");
```
That makes LLM-authored speech portable โ let the model write tags and whatever provider is wired up
does the right thing. Use `AnnotationHandling = SpeechAnnotationHandling.Preserve` when the text
legitimately contains square brackets, or `Strip` to remove them unconditionally.
| Provider | Mechanism | Notes |
| --- | --- | --- |
| ElevenLabs | Inline audio tags | Only on `eleven_v3`; capabilities derive from `TextToSpeechModel`, so older models strip |
| Typecast | `emotion_preset` + `emotion_intensity` | Preset vocabulary is small, so the mapping is lossy; `TypecastConfig.Emotion` is the fallback |
| Azure | `mstts:express-as style` + `styledegree` | Style support is per-voice; unsupported styles render in the default delivery |
| OpenAI | `instructions` | Needs an instruction-aware model such as `gpt-4o-mini-tts` |
| iOS / Android / Windows / Browser | none | Annotations stripped, tone discarded |
Custom providers report what they support via `ITextToSpeechProvider.ToneCapabilities` (defaulting
to `SpeechToneCapabilities.None`) and call `SpeechAnnotations.Resolve(text, options, ToneCapabilities)`
to get back the text and tone to actually use.
### Playing Audio (`IAudioPlayer`)
`IAudioPlayer` plays a `Stream`, or โ via `PlayAsync(string)` โ a **remote URL or a local file path**.
You pass a plain URL/path; each platform resolves it natively, so you never build a platform-specific
file URI:
```csharp
public class Player(IAudioPlayer audioPlayer)
{
public Task PlayRemote() => audioPlayer.PlayAsync("https://example.com/clip.mp3");
public Task PlayLocal() => audioPlayer.PlayAsync(Path.Combine(FileSystem.AppDataDirectory, "chime.mp3"));
public Task PlayStream(Stream mp3) => audioPlayer.PlayAsync(mp3);
}
```
An absolute `http`/`https` value is treated as a remote source (progressively streamed on Android,
Windows, and Browser; buffered on Apple); anything else is treated as a local file path. In the
browser, a local path means an app-relative URL (there is no device file system).
**Clips play concurrently.** Starting one does not stop the others, so background music, a sound
effect and a voice line can overlap. `StartAsync` hands back an `IAudioPlayback` you can stop on its
own โ `StopAsync()` on the player stops everything:
```csharp
public class Mixer(IAudioPlayer audioPlayer)
{
IAudioPlayback? music;
public async Task StartMusic()
=> this.music = await audioPlayer.StartAsync("https://example.com/theme.mp3");
// Overlaps the music instead of interrupting it
public Task Ping() => audioPlayer.PlayAsync(Path.Combine(FileSystem.AppDataDirectory, "ping.mp3"));
public Task FadeOutMusic() => this.music?.StopAsync() ?? Task.CompletedTask; // effects keep playing
public Task Silence() => audioPlayer.StopAsync(); // everything stops
}
```
`PlayAsync` is just `StartAsync` plus a wait on `IAudioPlayback.Completion`; a `CancellationToken`
passed to either one stops that clip and nothing else. `IAudioPlayer.Active` lists what is playing
right now.
### VU Meters (Audio Levels)
Both directions are metered, on the same normalized `0.0`โ`1.0` scale (dBFS mapped from a -50 dB
noise floor, so voice actually moves the bar).
**Outgoing โ playback / TTS.** `ITextToSpeechService` and `IAudioPlayer` raise `AudioLevelChanged`
while audio is playing. Gate UI on `IsPlayerAnalysisSupported`.
```csharp
if (tts.IsPlayerAnalysisSupported)
tts.AudioLevelChanged += (s, level) =>
MainThread.BeginInvokeOnMainThread(() => SpeakingBar.Progress = level);
```
**Incoming โ microphone.** `ISpeechToTextService` raises `InputLevelChanged` while listening โ gate
UI on `IsInputAnalysisSupported`. The lower-level `IAudioSource.InputLevelChanged` (raw capture) and
`IAudioMonitor.InputLevelChanged` (live mic-to-output) emit the same signal.
```csharp
if (stt.IsInputAnalysisSupported)
stt.InputLevelChanged += (s, level) =>
MainThread.BeginInvokeOnMainThread(() => ListeningBar.Progress = level);
```
| Surface | iOS / macOS | Android | Windows | Browser | Linux |
|---|---|---|---|---|---|
| Native `ITextToSpeechService` | โ
| โ
| โ | โ | n/a |
| Cloud `ITextToSpeechService` (Azure / OpenAI / ElevenLabs / custom) | โ
| โ
| โ | โ | โ
|
| `IAudioPlayer` (generic playback) | โ
| โ
| โ | โ | โ
|
| Cloud `ISpeechToTextService` (Azure / OpenAI / ElevenLabs / custom) | โ
| โ
| โ
| โ
| โ
|
| Native `ISpeechToTextService` | โ
| โ
| โ | โ | n/a |
| On-device `ISpeechToTextService` (Whisper) | โ | โ | โ | โ | โ
|
| `IAudioSource` (raw capture) | โ
| โ
| โ
| โ
| โ
|
| `IAudioMonitor` (live monitor) | โ
| โ
| n/a | n/a | โ
|
Cloud recognition meters the `IAudioSource` feeding the provider, so it works everywhere. Native
recognition depends on the platform: Apple taps the recognizer's own input node, Android reports the
`SpeechRecognizer` RMS callback, while Windows' and the browser's recognizers own the mic and expose
no level at all. Linux has no OS speech engine to wrap, so there is no native STT/TTS there at all โ
but every cloud provider works, `Shiny.Speech.Linux.Whisper` covers offline recognition, and playback
metering is supported because the Linux player decodes to PCM in managed code and sees every sample.
On Apple platforms, native TTS routes `AVSpeechSynthesizer` through `AVAudioEngine` +
`AVAudioPlayerNode` so audio levels can be tapped. The engine is created lazily on first speak and
kept warm across utterances โ only the first utterance pays ~50โ150 ms additional startup.
Levels are raised off the UI thread and throttled to ~20/sec on the capture side; marshal before
binding. Reset your bound value to `0` when playback/listening ends so the meter drains.
### Volume
`IAudioPlayer` exposes `Volume` (the device media volume, normalized `0.0`โ`1.0`), a
`VolumeChanged` event, and `IsVolumeControlSupported`. **Reading works on every platform; setting is
platform-limited** โ always guard a set with `IsVolumeControlSupported`.
```csharp
// Read anywhere
var level = player.Volume;
// Set only where supported (throws NotSupportedException on iOS / Mac Catalyst)
if (player.IsVolumeControlSupported)
player.Volume = 0.5f;
// Observe changes: hardware buttons, the OS volume UI, or a successful set
player.VolumeChanged += (_, v) =>
MainThread.BeginInvokeOnMainThread(() => MyVolumeSlider.Value = v);
```
| Platform | Read | Set | `VolumeChanged` source | Backing API |
|---|---|---|---|---|
| Android | โ
| โ
| System settings observer | `AudioManager` `STREAM_MUSIC` |
| Windows | โ
| โ
| Endpoint volume callback | WASAPI `IAudioEndpointVolume` (default render endpoint) |
| macOS | โ
| โ
* | CoreAudio property listener | HAL virtual main volume of the default output device |
| iOS / Mac Catalyst | โ
| โ | KVO on `outputVolume` | `AVAudioSession.OutputVolume` (read-only) |
| Browser (WASM) | โ
| โ
| Echoed on set | `HTMLAudioElement.volume` (app-local, **not** the OS volume) |
| Linux (PulseAudio / PipeWire) | โ
| โ
| Server subscription | Default sink volume via `pa_context_set_sink_volume_by_name` |
| Linux (ALSA fallback) | โ
โ | โ | n/a | n/a โ binding `snd_mixer` isn't worth it for the fallback path |
\* macOS is settable when the current default output device exposes a settable virtual main volume
(most built-in / USB devices do; some HDMI / aggregate devices don't) โ this is reflected by
`IsVolumeControlSupported`.
โ On the ALSA fallback `Volume` reads back the last value set in-process (defaulting to `1.0`) rather
than a real device level, and `IsVolumeControlSupported` is `false` so the setter throws. Check it
before assigning, as on iOS.
On device platforms `Volume` is the **system media volume** the hardware buttons control, independent
of any per-request TTS volume. On iOS / Mac Catalyst there is no supported OS API to change the system
volume, so the setter throws โ let the user adjust it via the hardware buttons or an `MPVolumeView`.
Browsers sandbox the OS volume, so there `Volume` is the app's own media-element volume (settable, and
it persists across plays).
### Capture Effects & Recording
`Shiny.Audio` can apply real-time DSP to the microphone and record the result to a WAV file. Effects
are pure managed code running on the normalized 16 kHz mono PCM every platform already produces, so
they behave identically on iOS, Android, Windows, Linux and the browser โ including pitch shift,
which no platform offers natively on capture.
#### Controlling effects live
An effect is an object you own. Build a chain, hand it to the session, then keep the reference and
mutate it โ every change is picked up on the next audio buffer.
```csharp
using Shiny.Audio;
var chain = new AudioEffectChain();
var pitch = chain.Add(new PitchShiftEffect { Semitones = 0 });
var echo = chain.Add(new EchoEffect { DelayMs = 250, Mix = 0.35f, Enabled = false });
var stream = await audio.Source.StartCaptureAsync(new AudioCaptureOptions { Effects = chain });
// ...all of this applies mid-capture, from any thread:
pitch.Semitones = 5; // change a value
echo.Enabled = true; // switch one effect on
chain.Enabled = false; // master bypass
```
There are three levels of on/off โ `chain.Enabled`, `effect.Enabled`, and each effect's `Mix` โ and
every numeric parameter is live-settable. Parameters are ramped and bypass is crossfaded, so moving a
slider or flipping a switch during a take doesn't click.
| Effect | Key parameters |
| --- | --- |
| `GainEffect` | `Gain`, `GainDb` |
| `NoiseGateEffect` | `ThresholdDb`, `AttackMs`, `ReleaseMs` |
| `BiquadFilterEffect` | `Type` (LowPass/HighPass/BandPass/Notch), `Frequency`, `Q` |
| `DistortionEffect` | `Drive`, `Mix` |
| `RingModEffect` | `Frequency`, `Mix` โ the robot voice |
| `EchoEffect` | `DelayMs`, `Feedback`, `Mix` |
| `ChorusEffect` | `RateHz`, `DepthMs`, `Mix`, `Feedback` |
| `ReverbEffect` | `RoomSize`, `Damping`, `Mix` |
| `PitchShiftEffect` | `Semitones` (ยฑ24) |
`AudioEffectPresets.Create(AudioEffectPreset.Robot)` builds ready-made chains โ `Robot`, `Chipmunk`,
`DeepVoice`, `Cathedral`, `Telephone`, `Megaphone`, `Ensemble` โ which are ordinary chains you can
keep adjusting.
> **Never put effects on audio bound for speech recognition.** Pitch, reverb and the rest destroy
> recognition and wake-word accuracy. This is the same hazard `AudioProcessingOptions.Analysis`
> warns about, one step worse.
#### Recording
`IAudioRecorder` owns the capture session and the drain loop, and writes 16 kHz mono PCM16 WAV.
```csharp
var recorder = audio.Recorder;
if (await recorder.RequestAccess() != AccessState.Available)
return;
recorder.InputLevelChanged += (_, level) => { /* meter reflects the recorded signal */ };
await recorder.StartAsync(new AudioRecordingOptions
{
Path = null, // null โ timestamped file under the app's local data dir
Mode = AudioRecordMode.Both, // Wet (default) | Dry | Both
Effects = chain
});
// ... adjust the chain live while it records ...
var recording = await recorder.StopAsync(); // null if nothing was captured
Console.WriteLine($"{recording!.Duration} โ {recording.Path} (dry: {recording.DryPath})");
```
`Mode` decides what is written: `Wet` is the processed take, `Dry` is the untouched microphone, and
`Both` writes two files so you can compare them. Recording always captures dry from the source and
applies the chain in the recorder's own loop โ which is what makes `Both` possible without splitting
the capture stream.
Because `Dry` keeps the clean take, you can re-render it later with different settings instead of
asking anyone to perform it again:
```csharp
AudioEffectProcessor.ProcessFile(recording.DryPath!, "take-cathedral.wav",
AudioEffectPresets.Create(AudioEffectPreset.Cathedral));
```
`WavWriter` and `WavReader` are public if you want to stream PCM to and from WAV yourself.
> **Format:** WAV/PCM16 only. AAC/MP3 would need a native encoder per platform (and Linux has none),
> so it is deliberately out of scope.
#### Latency and cost
Everything except pitch shift is cheap enough to be irrelevant at 16 kHz mono โ fine on a Raspberry
Pi. `PitchShiftEffect` adds up to one window (~50 ms) of latency, and beyond roughly ยฑ7 semitones a
voice starts to sound obviously processed. Reverb and pitch on a narrowband mono mic sound thinner
than they would on full-bandwidth stereo: these are voice effects, not mastering tools.
### Microphone Monitor & Routes (`IAudio`)
Inject the single `IAudio` facade to reach the whole audio surface โ `Player`, `Source`, `Monitor`,
`Devices`, `Recorder`, `Duplex` โ or keep injecting the focused interfaces directly. `IAudioMonitor` routes the mic straight
to the current output in near-real-time (a PA / "talk over a Bluetooth speaker"); `IAudioDevices`
reports which routes are active.
```csharp
using Shiny.Audio;
using Shiny; // AccessState
public class PaController(IAudio audio)
{
public async Task Start()
{
if (await audio.Monitor.RequestAccess() != AccessState.Available)
return;
audio.Monitor.InputLevelChanged += (_, level) => { /* drive a VU bar */ };
// No Processing โ routes to a Bluetooth A2DP speaker (phone mic + BT output).
await audio.Monitor.Start(new AudioMonitorOptions { Gain = 1.0 });
}
// Display where audio is flowing (e.g. "JBL Flip ยท BluetoothA2dp").
public string Output => $"{audio.Devices.CurrentOutput?.Name} ยท {audio.Devices.CurrentOutput?.Type}";
public Task Stop() => audio.Monitor.Stop();
}
```
`IAudioMonitor` is implemented on **iOS/Mac Catalyst and Android**, and `IAudioDevices` on **iOS/Mac
Catalyst, macOS, Android and Windows** (Linux registers both through `AddLinuxAudio()`); the facade
throws `PlatformNotSupportedException` if you touch them elsewhere.
#### Recording from a chosen microphone
Hand any input from `GetInputs()` to the capture or the recording; `null` (the default) is the
system's default input, which `CurrentInput` reports. A microphone unplugged since it was listed falls
back to the default rather than failing.
```csharp
var mic = audio.Devices.GetInputs().First(x => x.Type == AudioDeviceType.Usb);
await recorder.StartAsync(new AudioRecordingOptions { InputDevice = mic });
var stream = await audio.Source.StartCaptureAsync(new AudioCaptureOptions { InputDevice = mic });
```
It is honoured on macOS, Windows and Linux; on iOS/Mac Catalyst and Android it becomes the preferred
input, and the route follows.
#### Classifying the route
`AudioDevice.Type` normalizes the OS route into `AudioDeviceType` โ including wired: `WiredHeadphones`
(output only) and `WiredHeadset` (output **plus** mic). Rather than matching every variant by hand, use
the classification helpers:
```csharp
var output = audio.Devices.CurrentOutput;
if (output?.IsWired() == true) { /* 3.5mm jack, Lightning, or USB-C */ }
if (output?.IsBluetooth() == true) { /* HFP/SCO or A2DP */ }
if (output?.IsBuiltIn() == true) { /* phone speaker or earpiece โ nothing attached */ }
// "Is audio private to the user?" โ e.g. before speaking a TTS response out loud.
if (output?.IsHeadphones() == true)
await audio.Player.Play(reply);
// Will capture come from the accessory, or fall back to the phone mic?
var accessoryMic = output?.HasMicrophone() == true;
```
Each helper also exists on `AudioDeviceType` directly. Subscribe to `IAudioDevices.Changed` to react to
plug/unplug โ it fires on both platforms (iOS route-change notification, Android device callback).
- **`IsWired()` includes `Usb`.** On handsets with no 3.5mm jack the wired option *is* USB-C, and neither
platform reports those as a `Wired*` type (Android says `UsbHeadset`/`UsbDevice`, iOS says
`PortUsbAudio`). The trade-off: a USB audio interface or DAC also answers true. Use `IsHeadphones()`
when you specifically mean something worn on the head, accepting that it misses USB-C earbuds.
- **Bluetooth can't be narrowed.** A paired A2DP route is equally a set of earbuds or a room speaker;
no platform API distinguishes them, so `IsHeadphones()` counts all Bluetooth as private.
- **Bluetooth speaker vs. echo cancellation (iOS):** enabling `AudioProcessingOptions` forces Bluetooth
onto the low-quality HFP call profile, so a Bluetooth *speaker* (A2DP) drops back to the phone. Leave
processing **off** to reach a BT speaker; turn it on only for phone-speaker output where feedback is a
problem. **AirPlay/HomePod is not supported for a live mic** โ use Bluetooth for a wireless PA.
- **Device selection is Android-first:** Android enumerates/selects both input and output; iOS can
select the input but treats **output as observe-only** (AirPlay/Bluetooth output routing is owned by
the system picker). Use `CurrentInput`/`CurrentOutput` as a **display** property everywhere.
See the **Microphone** tab in the MAUI sample for a full page.
### Full-Duplex Voice for Calls (`IAudioDuplex`)
`IAudioDuplex` runs microphone capture and live PCM playback on **one** voice-processing session, so
the platform's echo canceller removes what the speaker plays from what the mic hears. Use it for a
phone / FaceTime-style call where PCM frames arrive from the network every ~20 ms and must play
straight away. `IAudioSource` plus `IAudioPlayer` can't do this: two sessions can't cancel each
other's echo, and `IAudioPlayer.StartAsync(Stream)` is built for clips (Apple buffers the whole stream,
Android copies it to a file first).
```csharp
using Shiny.Audio;
public class CallAudio(IAudio audio) // or inject IAudioDuplex directly
{
IAudioDuplexSession? session;
public async Task Start(INetworkPipe pipe, CancellationToken ct)
{
if (await audio.Duplex.RequestAccess() != AccessState.Available)
return;
session = await audio.Duplex.StartAsync(new AudioDuplexOptions
{
SampleRate = 16000, // or 8000 for narrow links
Speakerphone = true // loudspeaker rather than the earpiece
}, ct);
// Mic โ network: one 20 ms PCM16 mono frame per item, already echo-cancelled.
_ = Task.Run(async () =>
{
await foreach (var frame in session.Captured.ReadAllAsync(ct))
await pipe.Send(frame, ct);
}, ct);
// Network โ speaker: hand over whatever arrives; it's jitter-buffered.
pipe.Received += pcm => session.Play(pcm.Span);
}
public void ToggleSpeaker() => session!.Speakerphone = !session.Speakerphone; // applies live
// Pick a mic from audio.Devices.GetInputs(); null goes back to the system default. Applies live.
public void UseMic(AudioDevice? mic) => session!.InputDevice = mic;
public Task Stop() => session?.StopAsync() ?? Task.CompletedTask; // restores the audio session
}
```
| `AudioDuplexOptions` | Default | |
|---|---|---|
| `SampleRate` | `16000` | `8000` or `16000`, both directions |
| `Processing` | `VoiceChat` | AEC + NS + AGC. Null / nothing enabled = no echo cancellation |
| `Speakerphone` | `true` | Loudspeaker rather than the earpiece (a headset still wins) |
| `InputDevice` | `null` | The mic, from `IAudioDevices.GetInputs()`. Null = the system default. Changeable live via `IAudioDuplexSession.InputDevice` |
| `FrameDuration` | 20 ms | Size of each `Captured` item, whole ms 10โ100 |
| `PlaybackLatency` | 60 ms | Jitter-buffer target: playback waits for this much, and re-primes after an underrun |
| `MaxPlaybackLatency` | 200 ms | When more is queued, the oldest audio is dropped back to `PlaybackLatency` |
How the playback queue behaves:
- **`Play` is thread-safe and never blocks.** Any length; gaps (late or lost packets) play as silence.
- **Priming:** nothing plays until about `PlaybackLatency` is queued; after the queue runs dry it primes
again, so one late packet doesn't turn into a stutter on every frame after it.
- **Overflow:** a burst that takes the queue past `MaxPlaybackLatency` drops the oldest audio back to
`PlaybackLatency`, so delay can't build up over a call. `Buffered` reports what's queued.
- **`Captured` is bounded** (about a second). A reader that falls behind loses the oldest frames.
Choosing the microphone (`InputDevice`):
- **Set it live.** `session.InputDevice = mic` switches without stopping the call; `null` goes back to
the default. An output device throws `ArgumentException`. A mic that isn't connected falls back to the
default, and `InputDevice` then reads `null`.
- **A headset's mic and speaker move together** (iOS, Mac Catalyst, Android): choosing a Bluetooth, wired
or USB headset mic routes the call's audio to that headset too, and `Speakerphone` has no effect while
it's chosen. Choosing the **built-in** mic while a headset is connected takes the call off the headset,
to the loudspeaker or earpiece per `Speakerphone`. With `null`, a connected headset wins as before.
- **`InputDeviceChanged`** fires when the platform changes it underneath you: the chosen mic was
unplugged or disconnected, the session is back on the default, and `InputDevice` reads `null`. It
isn't raised for your own sets.
| Platform | How |
|---|---|
| iOS / Mac Catalyst | `AVAudioSession` `PlayAndRecord` + `VoiceChat` mode (`DefaultToSpeaker` for speakerphone, HFP Bluetooth when `AllowBluetooth`), one `AVAudioEngine` with voice processing on its input node, an `AVAudioConverter` tap and an `AVAudioPlayerNode` fed from the jitter buffer. Rebuilds itself on a route change. `InputDevice` is the session's preferred input (`SetPreferredInput`); the route change it causes rebuilds the engine, so expect a short blip |
| macOS | The same engine, without `AVAudioSession`; `Speakerphone` has no effect. `InputDevice` points the voice-processing unit at that CoreAudio device; switching live restarts the engine (a gap of a few tens of ms, queued playback kept). Only the input changes: macOS output is a separate device |
| Android | `AudioRecord` (`VOICE_COMMUNICATION`) + streaming `AudioTrack` (`USAGE_VOICE_COMMUNICATION`) on one audio session, `AcousticEchoCanceler` / `NoiseSuppressor` / `AutomaticGainControl` when the device has them, `MODE_IN_COMMUNICATION` and transient audio focus while running. Speakerphone through `setCommunicationDevice` (API 31+) or `setSpeakerphoneOn` below it. `InputDevice` is the record's `setPreferredDevice`, switched in place, and steers the communication device as above. Below API 31 the duplex doesn't start a Bluetooth SCO link, so a Bluetooth headset mic may not be reachable there. Needs `MODIFY_AUDIO_SETTINGS` in the manifest |
| Windows, Linux, Browser | Not registered: `IAudio.Duplex` throws `PlatformNotSupportedException` |
- **One session at a time.** A second `StartAsync` while one is running throws `InvalidOperationException`.
- **Stop restores** the audio session category / mode (Apple) or audio mode, routing and focus (Android)
that the session changed. Disposing the session stops it.
- Echo-cancellation quality is the platform's and the device's: it's very good on iPhones and varies
across Android hardware. Without processing, use a headset.
### Speech-to-Text
The `ISpeechToTextService` uses a Start/Stop model with events, allowing multiple consumers to observe recognition results simultaneously.
```csharp
public class MyService(ISpeechToTextService stt) : IDisposable
{
public async Task StartListeningAsync()
{
var access = await stt.RequestAccess();
if (access != AccessState.Available)
return;
// Subscribe to events
stt.ResultReceived += OnResult;
stt.KeywordHeard += OnKeyword;
stt.Error += OnError;
// Start listening (throws if already listening)
await stt.Start(new SpeechRecognitionOptions
{
Culture = CultureInfo.GetCultureInfo("en-US"),
SilenceTimeout = TimeSpan.FromSeconds(3),
Keywords = ["Yes", "No", "Maybe"]
});
}
public async Task StopListeningAsync()
{
await stt.Stop(); // no-op if not listening
stt.ResultReceived -= OnResult;
stt.KeywordHeard -= OnKeyword;
stt.Error -= OnError;
}
void OnResult(object? sender, SpeechRecognitionResult result)
=> Console.WriteLine($"[{(result.IsFinal ? "FINAL" : "partial")}] {result.Text}");
void OnKeyword(object? sender, string keyword)
=> Console.WriteLine($"Keyword detected: {keyword}");
void OnError(object? sender, SpeechRecognitionError error)
=> Console.WriteLine($"Error: {error.Message}");
public void Dispose() => StopListeningAsync().GetAwaiter().GetResult();
}
```
#### Continuous sessions and recovery
`Start()` opens the microphone and keeps it open. Both native recognizers are single-utterance
underneath, so the service re-arms them for you after every final result โ you start once and stay
listening until you call `Stop()`.
Transient failures in that loop (the mic taken by another capture, a busy recognizer, a dropped
network round trip) no longer end the session. The service reports them through `Error` and re-arms
behind a backoff โ 250ms, then doubling to a 4 second ceiling โ resetting the moment a result comes
back. After `SpeechRetryPolicy.MaxConsecutiveFailures` (5) failures in a row it stops the session
rather than looping, so `IsListening` never reports `true` for a session that has quietly died.
Errors that retrying cannot fix โ a missing permission, an unsupported language โ stop immediately.
#### On-device recognition
`PreferOnDevice` asks for recognition with no network round trip and no session length cap, which is
what long continuous sessions and offline use want:
```csharp
await stt.Start(new SpeechRecognitionOptions { PreferOnDevice = true });
```
It is best-effort on every platform. iOS sets `RequiresOnDeviceRecognition` when the locale supports
it; Android uses the on-device recognizer when the device has one installed (API 31+) and otherwise
falls back to the system recognizer with the offline hint set. A device without local recognition
stays on the network path rather than failing.
### Voice Processing (Noise Suppression & Echo Cancellation)
Microphone capture can request platform voice-processing effects to strip background noise and,
critically, to **cancel your text-to-speech output from the mic** so it isn't re-captured while the
mic is open (barge-in). Configure it via `AudioProcessingOptions` โ either directly on
`IAudioSource.StartCaptureAsync(...)` or through `SpeechRecognitionOptions.AudioProcessing`, which is
honored wherever the capture belongs to Shiny: every cloud provider (they record through
`IAudioSource`) and the **native iOS / Mac Catalyst / macOS recognizer**, which owns its own
`AVAudioEngine`. Leaving it `null` keeps the default those paths have always used โ the full
`VoiceChat` chain โ rather than raw capture.
> **Ignored on Android.** Recognition there runs in the platform's own out-of-process
> `SpeechRecognizer` service, which opens the microphone itself and exposes no voice-processing
> controls, so there is no capture session to apply this to. Setting it logs a warning. Use a cloud
> provider if the recognition path needs these effects on Android.
```csharp
await stt.Start(new SpeechRecognitionOptions
{
Culture = CultureInfo.GetCultureInfo("en-US"),
AudioProcessing = AudioProcessingOptions.VoiceChat // AEC + noise suppression + AGC
});
// or, capturing raw audio directly:
var stream = await audioSource.StartCaptureAsync(new AudioProcessingOptions
{
EchoCancellation = true, // subtracts speaker/TTS output from the mic signal
NoiseSuppression = true, // attenuates steady background noise
AutomaticGainControl = true // normalizes capture level
});
// capture for a *model* (speaker recognition, wake words) rather than a listener:
var raw = await audioSource.StartCaptureAsync(AudioProcessingOptions.Analysis);
```
`AudioProcessingOptions.Analysis` is `None` plus `AllowBluetooth = false`, and it matters whenever the
audio feeds an embedding model. The effects above are adaptive and exist to normalize away speaker and
channel characteristics โ the very thing such a model measures โ so two recordings of one person come
back different; and a Bluetooth mic runs over HFP at 8 kHz narrowband, which quietly makes the captured
bandwidth depend on what is paired. On iOS/Mac Catalyst, requesting no effects puts the session in
`Measurement` mode (minimum system input processing) rather than `VoiceChat`.
Each flag is **best-effort** and maps to native voice processing:
| Effect | iOS / macOS | Android | Windows | Browser |
|---|---|---|---|---|
| Echo Cancellation | โ
Voice-Processing I/O | โ
`AcousticEchoCanceler` + VoiceCommunication | โ ๏ธ best-effort (Communications pipeline) | โ
WebRTC AEC3 |
| Noise Suppression | โ
(bundled) | โ
`NoiseSuppressor` | โ ๏ธ best-effort | โ
|
| Automatic Gain Control | โ
(bundled) | โ
`AutomaticGainControl` | โ ๏ธ best-effort | โ
|
Notes:
- **Apple** bundles all three into a single Voice-Processing I/O unit โ enabling any flag enables the whole chain (they can't be toggled independently).
- **Android** effect availability is device/driver dependent; unavailable effects are skipped. Requesting echo cancellation also routes capture through `VoiceCommunication`.
- **Windows** `AudioGraph` exposes no per-effect control; requesting any effect selects the `Communications` capture category, which engages driver-provided AEC/NS when present.
- These are **OS/hardware** cancellers referencing the real speaker feed, so any device audio (not just library-played TTS) is cancelled.
- **Native on-device** `ISpeechToTextService` implementations manage their own microphone and are unaffected by this setting โ it applies to `IAudioSource` capture (cloud providers, raw capture).
### Extension Methods (Convenience)
```csharp
// Simple: wait for silence (starts and stops automatically)
var text = await stt.ListenUntilSilence(cancellationToken: ct);
// Wake word: "Hey Computer, do something" โ returns "do something"
var command = await stt.StatementAfterKeyword(["Hey Computer"], cancellationToken: ct);
// Wait for a specific keyword (with optional timeout)
var answer = await stt.WaitListenForKeywords(["Yes", "No"], timeout: TimeSpan.FromSeconds(30), cancellationToken: ct);
// Continuous keyword stream
await foreach (var keyword in stt.ListenForKeywords(["Up", "Down", "Left", "Right"], cancellationToken: ct))
{
Console.WriteLine($"Direction: {keyword}");
}
```
## Custom Cloud Provider
Implement `ISpeechToTextProvider` and/or `ITextToSpeechProvider` from `Shiny.Speech.Cloud`:
```csharp
public class MyCloudSttProvider : ISpeechToTextProvider
{
public event EventHandler<SpeechRecognitionError>? Error;
public async IAsyncEnumerable<SpeechRecognitionResult> RecognizeAsync(
Stream audioStream,
SpeechRecognitionOptions? options = null,
CancellationToken cancellationToken = default)
{
// Read PCM audio from audioStream (16kHz, 16-bit, mono)
// Yield recognition results...
// For continuous recognition, surface non-fatal errors (e.g. a transient
// network blip between chunked requests) without aborting the session:
// Error?.Invoke(this, new SpeechRecognitionError("network blip", ex));
}
}
// Register:
builder.Services.AddCloudSpeechToText<MyCloudSttProvider>();
```
`CloudSpeechToText` subscribes to the provider's `Error` event and forwards it to the service-level `ISpeechToTextService.Error`, so app code only needs to wire one handler.
## Platform Requirements
| Platform | STT | TTS | Audio Capture | Audio Playback |
|----------|-----|-----|---------------|----------------|
| iOS 15+ (incl. CarPlay) | SFSpeechRecognizer | AVSpeechSynthesizer | AVAudioEngine | AVAudioPlayer |
| Android 26+ | SpeechRecognizer | Android TTS | AudioRecord | MediaPlayer |
| Windows 10 19041+ | Windows.Media.SpeechRecognition | Windows.Media.SpeechSynthesis | AudioGraph | MediaPlayer |
| Browser (WASM) | Web Speech API (`SpeechRecognition`) | Web Speech API (`SpeechSynthesis`) | Web Audio API (`getUserMedia` + `ScriptProcessorNode`) | HTML5 `Audio` |
| Linux (`Shiny.Audio.Linux`) | Whisper on-device (`Shiny.Speech.Linux.Whisper`) or any cloud provider | โ native โ use a cloud provider | PulseAudio / PipeWire (`pa_simple`), ALSA fallback | PulseAudio / PipeWire (`pa_simple`), ALSA fallback |
### Linux
Linux support lives in a separate **`Shiny.Audio.Linux`** package. It provides all four audio
services โ capture, playback, device enumeration and live mic monitoring โ over PulseAudio/PipeWire,
falling back to ALSA where no sound server is running (headless boxes, minimal containers, Raspberry
Pi images).
```csharp
using Shiny;
// Call this BEFORE AddSpeechServices() / AddCloudSpeechToText<T>() โ those use TryAdd,
// so whatever is registered first wins. It is a no-op when not running on Linux, so it
// is safe to leave in shared startup code.
builder.Services.AddLinuxAudio();
builder.Services.AddCloudSpeechToText<AzureSpeechToTextProvider>();
builder.Services.AddCloudTextToSpeech<AzureTextToSpeechProvider>();
```
**There is no native Linux STT/TTS** โ Linux has no OS speech engine to wrap the way iOS, Android and
Windows do, so `AddSpeechToText()` / `AddTextToSpeech()` register nothing there. Use any of the cloud
providers (Azure, OpenAI, ElevenLabs, Typecast, Microsoft.Extensions.AI); they are pure HTTP over the
PCM stream and work unchanged once `AddLinuxAudio()` has supplied an `IAudioSource`. For recognition
without a cloud account at all, see **Whisper on-device** below.
Notes:
- **No resampler ships in this library.** Streams are opened at exactly the format required (16 kHz
mono for capture, a decoded clip's native rate for playback); PulseAudio converts server-side and
ALSA's `default` device converts through its `plug` plugin.
- **Playback decodes in managed code.** Linux has no system media decoder to hand a stream to, so
MP3 (what Azure, OpenAI and ElevenLabs return by default) is decoded with `NLayer`, and WAV/PCM
directly. Other containers throw `NotSupportedException` naming the format.
- **Voice processing is best-effort.** PulseAudio exposes echo cancellation as a virtual source from
`module-echo-cancel`; when it is loaded, `AudioProcessingOptions` with `EchoCancellation` selects
it, otherwise capture is raw.
- **Runtime dependencies** are the distro's own: `libpulse-simple.so.0` (usually `libpulse0`) or
`libasound.so.2` (usually `libasound2`). Check `LinuxAudioServiceCollectionExtensions.IsLinuxAudioAvailable`
at startup to fail fast where neither is present.
- `IAudioDevices.ShowOutputPicker()` is a no-op โ Linux has no system route picker equivalent to
iOS's `AVRoutePickerView`; routing lives in the desktop environment's sound settings. `Changed`
fires on PulseAudio/PipeWire only (ALSA has no change notification).
### Whisper on-device (Linux)
**`Shiny.Speech.Linux.Whisper`** runs OpenAI's Whisper locally through
[whisper.cpp](https://github.com/ggerganov/whisper.cpp) (via
[Whisper.net](https://github.com/sandrohanea/whisper.net)) โ the closest thing Linux has to the
native recognizers iOS, Android and Windows ship with. No cloud account, no API key, no network at
runtime, no per-minute billing.
```csharp
using Shiny;
using Shiny.Speech.Linux;
using Whisper.net.Ggml;
builder.Services.AddLinuxAudio(); // must come first โ supplies IAudioSource
builder.Services.AddLinuxWhisperSpeechToText(GgmlType.BaseEn, QuantizationType.Q5_1);
// or with full control:
builder.Services.AddLinuxWhisperSpeechToText(new WhisperConfig
{
ModelType = GgmlType.BaseEn,
Quantization = QuantizationType.Q5_1,
InitialPrompt = "Shiny, MAUI, Blazor", // bias the decoder toward your vocabulary
SilenceRmsThreshold = 500 // VAD sensitivity
});
```
It registers a normal `ISpeechToTextService`, so `ListenUntilSilence()`, `StatementAfterKeyword()`
and the rest of the extension methods work unchanged. Registration is a **no-op off Linux**, so it is
safe to leave in shared startup code.
The model is downloaded from Hugging Face on first use and cached in
`~/.local/share/shiny.speech/whisper`. That first call costs a download plus several seconds of model
load, so call `PrepareAsync()` at startup rather than making the user's first utterance pay for it:
```csharp
var provider = host.Services.GetRequiredService<ISpeechToTextProvider>();
await ((WhisperSpeechToTextProvider)provider).PrepareAsync();
```
Notes:
- **Speech-to-text only.** Whisper is a recognition model; there is no Whisper TTS. Pair it with a
cloud TTS provider, or skip TTS entirely.
- **No partial results.** Whisper is a batch model with a 30-second window, not a streaming
recognizer. The provider runs client-side voice activity detection over the mic stream and turns
each speechโsilence segment into one inference and one `IsFinal = true` result โ the same shape as
the ElevenLabs Scribe provider. Tune with `SilenceRmsThreshold`, `MinUtteranceDurationMs` and
`MaxUtteranceDurationMs`.
- **Model sizing.** `Tiny`/`Base` are the realistic choices on a Raspberry Pi 4/5 โ expect roughly
realtime at `Base`, slower above `Small`. This suits push-to-talk and wake-word-then-command far
better than continuous dictation. On x64 desktops/servers, `Small` and up are comfortable. The
`*En` variants are meaningfully more accurate than the multilingual model of the same size when you
only need English.
- **Platform support is all mainstream Linux, not just Pi** โ `linux-x64`, `linux-arm64` and
`linux-arm` natives all ship. Requirements are `libstdc++6` and glibc 2.31+ (Debian 11+ / Ubuntu
20.04+ / Raspberry Pi OS Bullseye+). On **x86/x64 the CPU must support AVX, AVX2, FMA and F16C**;
older CPUs need an added `Whisper.net.Runtime.NoAvx` package reference. ARM has no such
requirement.
- **Trim the natives on publish.** The bundled `Whisper.net.Runtime` carries binaries for every
platform it supports; publishing with an explicit RID (`dotnet publish -r linux-arm64`) copies only
the ones you need.
- **GPU is opt-in.** Add `Whisper.net.Runtime.Cuda` (or `.Vulkan`) and set `UseGpu = true`; the CPU
runtime bundled here ignores the flag.
- **Hallucination guards are on by default.** Whisper emits `[BLANK_AUDIO]` and `(wind blowing)`-style
annotations for silence and noise; `FilterNonSpeechAnnotations` strips them and drops results that
filter down to nothing. `CarryContextBetweenUtterances` is off so a bad transcription can't poison
everything after it.
### Browser (Blazor WebAssembly)
No manifest changes and **no `<script>` tag** needed โ the browser prompts the user for microphone access automatically, and the JS interop module ships **inside the `Shiny.Audio` package** as a static web asset (`_content/Shiny.Audio/shiny-audio.js`). It is loaded on demand via `JSHost.ImportAsync`, so referencing `Shiny.Speech` (or `Shiny.Audio` directly) is all that's required.
> **Note:** `IAudioSource` captures raw PCM audio in the browser using the Web Audio API (`getUserMedia` + `ScriptProcessorNode`), downsampled to 16kHz 16-bit mono. Audio playback (`IAudioPlayer`) accepts any browser-supported format via a base64 data URL.
### iOS/macOS
Add to `Info.plist`:
```xml
<key>NSSpeechRecognitionUsageDescription</key>
<string>Speech recognition description</string>
<key>NSMicrophoneUsageDescription</key>
<string>Microphone description</string>
```
### Android
Add to `AndroidManifest.xml`:
```xml
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
```
`MODIFY_AUDIO_SETTINGS` is required for the TTS audio-level Visualizer, the native STT beep suppression and `IAudioDuplex` (call mode and speakerphone routing).
Nothing else is needed. `Shiny.Speech` ships its own library manifest carrying the `<queries>`
declaration that makes the platform recognition service visible on API 30+, and it merges into your
app automatically:
```xml
<queries>
<intent><action android:name="android.speech.RecognitionService" /></intent>
</queries>
```
Without it, package visibility filtering hides the recognition service, `IsSupported` is `false` and
`RequestAccess()` returns `AccessState.NotSupported` before the microphone permission is ever
requested. Permissions are deliberately *not* declared by the library โ `RECORD_AUDIO` is a dangerous
permission and would surface in the store listing of an app that only uses text-to-speech.
---
# Shiny.AiConversation
Session-based AI conversations for .NET apps on top of [Microsoft.Extensions.AI](https://devblogs.microsoft.com/dotnet/introducing-microsoft-extensions-ai/):
streamed replies, typed questions, persisted and resumable sessions, and token/context tracking.
[](https://www.nuget.org/packages/Shiny.AiConversation/)
> The package-level readme lives at [`src/aiconversation/readme.md`](src/aiconversation/readme.md).
> **v3 is a session-based rewrite** (breaking: `IAiConversationService`, `IMessageStore` and `IContextProvider`
> are gone). The core package no longer depends on Shiny.Speech or Shiny.Audio; voice lives in
> `Shiny.AiConversation.Voice`, the chat UI in `Shiny.AiConversation.Maui`.
## Features
- **Sessions** โ `IAiSessionManager` starts, resumes (`Get` / `GetLatest`), queries and deletes sessions. Sessions never end; they're persisted and can be picked up again at any time
- **Streaming turns** โ `session.Send(...)` returns an `IAiTurn` whose events stream text as it arrives; a new `Send` supersedes the active turn
- **Typed questions** โ the model calls the built-in `ask_user` tool when it needs an answer; the session moves to `AwaitingUser` with `PendingQuestions` (optionally with multiple-choice `AiChoice`s). A reply that ends in `?` still keeps the conversation open
- **Interruption** โ `turn.Interrupt(spokenText)` records what the user actually heard, and the model is told the reply was cut off
- **Usage & context** โ cumulative `SessionUsage` for billing and a `ContextSnapshot` for how full the context window is, with `Normal` / `Warning` / `Critical` pressure for the UI to act on
- **Interceptors** โ `ISessionInterceptor` adds tools and system prompts per session (`OnSessionLoaded`) or per turn (`OnBeforeTurn`), optionally by `Profile`
- **Honest persistence** โ the user message is stored before the model is called and the reply is stored as streaming then finalized, so failures and crashes leave an accurate record. Tool calls are persisted too
- **One event stream** โ `session.GetEvents()` is an `IAsyncEnumerable` that starts with a `Snapshot`, so a UI binds to one ordered stream
- **Compaction** โ sessions never end, so older messages are folded into a rolling summary: on demand with `session.Compact()`, or automatically as a safety net near the context limit. A failed summary degrades to a hard trim, never a failed turn
- **Assistant memory** โ a silent `memory` tool lets the AI keep notes across conversations (session, profile or global scope) without reading them aloud. Only an index goes into each prompt, limits are fed back to the model, and `manager.Memory` lets the user see, edit and delete everything. On by default
- **Conversation search** โ opt-in `search_conversations` / `read_conversation` tools find things in past (and compacted) conversations; plus optional session titles and long-term memory from recent sessions' summaries
- **SQLite persistence** โ `Shiny.AiConversation.Stores.SqliteDocDb` stores sessions, messages and memory with database-checked concurrency and full-text search
- **Voice** โ `Shiny.AiConversation.Voice` drives a session by voice: push-to-talk, optional wake word, follow-up answers without the wake word, interruption (quiet words or full barge-in) and sentence-by-sentence text-to-speech that starts before the reply is finished. Works with any Shiny.Speech provider
- **MAUI chat UI** โ `AiChatView` binds a `ChatView` to a session: streamed bubbles, choice chips, history paging, photos, a push-to-talk action and context warnings; `AiMemoryView` shows (and edits) what the assistant remembers
- **Images & camera** โ `AiInput.WithImage(...)` sends photos (stored as attachments, never inline); `Shiny.AiConversation.Maui.Camera` lets the assistant ask to use the camera (photo, or text/barcode/document scans) after asking the user
## Quick Start
```csharp
// Register an IChatClient in DI (from any Microsoft.Extensions.AI-compatible provider)
builder.Services.AddChatClient(new OpenAIClient("your-api-key").GetChatClient("gpt-4o").AsIChatClient());
builder.Services.AddShinyAiConversation(opts =>
{
opts.SetSqliteDocDbStores(Path.Combine(FileSystem.AppDataDirectory, "ai.db")); // default: in-memory
opts.AddInterceptor<MyInterceptor>(); // optional: tools + prompts
opts.AddTool(AIFunctionFactory.Create(GetWeather), AiToolFlags.None);
opts.AddSessionTitles(); // optional built-ins
opts.AddConversationSearch();
opts.AddLongTermMemory();
// opts.ConfigureMemory(m => m with { Enabled = false }); // memory is on by default
});
// Voice (register your Shiny.Speech providers yourself, e.g. AddSpeechServices() or a cloud provider)
builder.Services.AddShinyAiConversationVoice(voice => voice.AddVoiceSelectionTools());
// MAUI: let the assistant use the camera (needs .UseShinyCamera() on the app builder)
builder.Services.AddAiConversationCamera();
```
### Chat client providers
```csharp
// Ready-made static OpenAI-compatible provider (Shiny.AiConversation.OpenAi)
opts.AddStaticOpenAIChatClient(apiToken: "your-api-key", endpointUri: "https://api.openai.com/v1", modelName: "gpt-4o");
// MAUI GitHub Copilot โ self-contained device-code OAuth, tokens in SecureStorage (Shiny.AiConversation.Maui.GithubCopilot)
opts.AddGithubCopilotChatClient();
```
For other backends, implement `IChatClientProvider` and register with `opts.SetChatClientProvider<MyProvider>()`.
Set `ContextWindow` on the provider (or `AiSessionSettings.ContextWindow`) to get context pressure.
### Talk to the AI
```csharp
public class ChatViewModel(IAiSessionManager sessions)
{
IAiSession? session;
public async Task Send(string text)
{
this.session ??= await sessions.GetLatest() ?? await sessions.Start();
var turn = this.session.Send(AiInput.Text(text));
await foreach (var e in turn.Events)
{
if (e is TextDelta delta)
AppendToBubble(delta.Text);
}
var result = await turn.Completion; // never throws; check result.Outcome
if (result.Outcome == AiTurnOutcome.AwaitingUser)
ShowChoices(this.session.PendingQuestions);
}
// a tapped choice goes back as a normal message
public void Answer(AiQuestion question, AiChoice choice)
=> this.session!.Send(AiInput.Answer(question, choice.Id));
}
```
A UI that wants everything (both sides of the conversation, state, usage, errors) subscribes once:
```csharp
await foreach (var e in session.GetEvents(ct)) // first item is always a Snapshot
{
switch (e)
{
case MessageAdded m: ...; break;
case TurnEventOccurred { Event: TextDelta d }: ...; break;
case StateChanged s: ...; break;
case ContextPressureChanged p: ...; break; // e.g. "this conversation is getting long"
}
}
```
## API Overview
| Type | Description |
|------|-------------|
| `IAiSessionManager` | `Start(settings)`, `Get(id)`, `GetLatest()`, `Query(query)`, `Delete(id)`. Stateless: every load returns a new instance that the caller disposes |
| `IAiSession` | `Send(input)`, `GetEvents()`, `GetMessages(beforeSequence, count)`, `SetTitle`, `UpdateSettings`; `State`, `PendingQuestions`, `Usage`, `Context`, `ActiveTurn` |
| `IAiTurn` | `Events` (replayable), `Completion` (`AiTurnResult` with `Outcome`: Completed / AwaitingUser / Interrupted / Superseded / Failed), `Interrupt(spokenText)` |
| `AiInput` | `Text(...)`, `Voice(...)` (asks for spoken-style replies and at most one question), `Answer(question, choiceIds)` |
| `AiSessionSettings` | Persisted per session: `Profile`, `ContextWindow`, warning/critical thresholds, `Tags`, `Metadata`, `ExcludeFromSearch` |
| `ISessionInterceptor` | `OnSessionLoaded` (runs on Start **and** every Get), `OnBeforeTurn`, `OnAfterTurn` |
| `IAiSessionStore` | Session headers + append-only messages. `InMemoryAiSessionStore` is the default; `SetSqliteDocDbStores(path)` for SQLite |
| `IAiMemoryService` | `manager.Memory`: list, view, edit and delete what the assistant remembers ("forget everything" = `Clear()`) |
| `CompactionSettings` | Per session: `AutoCompactThreshold` (0.95, null disables), `KeepRecentMessages`, `MaxMessages`, `FlushToMemory` |
| `AiMemoryOptions` | `Enabled`, `DisabledProfiles` and the soft limits (document size, index size, scope size, writes per turn) |
| `IChatClientProvider` | Supplies the `IChatClient`; optional `ContextWindow`. The session adds function invocation itself when the client lacks it |
| `IVoiceConversation` | `Attach(session, options)`, `PushToTalk()`, `StopSpeaking()`, `Pause()` / `Resume()`, `Detach()`, `GetEvents()`; `State` (Off / WaitingForWakeWord / Listening / FollowUp / Thinking / Speaking / Paused) |
| `VoiceOptions` | `WakeWord` (off by default), `Interruption` (Off / QuietWords / Full), `QuietWords`, `FollowUpTimeout`, `Acknowledgement` (None / AudioBlip / LessWordy / Full) |
| `AiChatView` | MAUI `ChatView` bound to `Session` (or it resumes the latest session itself) |
## Dependencies
| Package | Purpose |
|---------|---------|
| [Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI) | `IChatClient` abstraction and the tool loop |
| Shiny.Speech (this repo) | Speech-to-text / text-to-speech for `Shiny.AiConversation.Voice` |
| [Shiny.Maui.Controls](https://shinylib.net/controls/) | `ChatView` and the camera modal for the MAUI packages |
## Samples
- [MAUI sample](samples/MauiSample) โ chat, voice (Aura), settings and assistant memory, sharing one session through a small `ConversationHost`
- [Blazor sample](samples/BlazorSample) โ the same over browser speech (deployed to https://shinyorg.github.io/speech/)
## License
MIT
## Commercial Support
Shipping this to production? [ShinySoft Technologies](https://shinysoft.biz), the company that builds and maintains the Shiny libraries, offers priority issue response, support agreements, architecture reviews and private office hours. See [Commercial Support](https://shinylib.net/foundation/support/?utm_source=github&utm_medium=readme#commercial-support) or [get in touch](https://shinysoft.biz/contact?utm_source=github&utm_medium=readme). Community support through GitHub issues and discussions stays free.