integrate-local-tts

name

integrate-local-tts

description

Maintain Voice Layer's local text-to-speech system across TypeScript reply chunking, Tauri IPC, retained sherpa-onnx adapters, Chat CPAL playback, and Live AVAudioEngine playback. Use when implementing, debugging, profiling, benchmarking, or documenting assistant speech; changing Supertonic or Pocket behavior; tuning sentence boundaries, native coalescing, ONNX threads, resampling, buffering, cancellation, drain semantics, or speech metrics; or validating Chat and Live TTS compatibility.

Integrate Local TTS

Preserve one request-scoped speech pipeline while respecting the different Chat and Live playback backends. Optimize from matched production evidence, not model reputation or an unrelated standalone demo.

Start from the current system

1. Read AGENTS.md and record git status --short --branch.
2. Read references/architecture.md completely before changing TTS code.
3. Read references/models-and-benchmarking.md completely when adapter behavior, performance, chunk sizing, progressive PCM, or benchmark claims are in scope.
4. Inspect the current source-of-truth values rather than copying tuning from this skill. Model pins, tuning, callback behavior, and playback implementations can change.
5. Preserve unrelated changes. Never download a model during startup, build, ordinary tests, or a cached-only benchmark.

Locate the change at its owner

• Change visible-delta chunk eligibility only in src/sentence-chunks.ts and its coordinator tests.
• Change request serialization, one-in-flight enqueue, successful final flush, or cancellation discard in src/assistant-coordinator.ts.
• Change Chat IPC/status handling in src/tts.ts; change spoken Live routing/state in src/live.ts. Typed Live turns must remain silent.
• Change native ordering, bounded admission, coalescing, finish/cancel, or terminal accounting in src-tauri/src/tts/stream.rs.
• Change output ownership, adapter routing, closed tuning, or Tauri commands in src-tauri/src/tts/mod.rs.
• Change Supertonic or Pocket loading and synthesis only in its adapter. Keep the retained actor and exact pinned artifacts.
• Change shared Chat/Live reply resampling in src-tauri/src/tts/reply_resampler.rs; change Chat CPAL buffering, rendering, and drain completion in src-tauri/src/tts/playback.rs.
• Change Live scheduling/backpressure in src-tauri/src/live/mod.rs; keep its voice-processing graph in src-tauri/src/live/audio_macos.rs.

Do not push model-specific runtime options into TypeScript. Do not send PCM over IPC, route Live playback through CPAL, or make Chat acquire microphone/VAD/ASR state.

Preserve latency and lifecycle invariants

• Start model preparation concurrently with LLM generation; never delay visible text on speech readiness.
• Synthesize the first eligible semantic chunk immediately. Never wait to fill a batch before first audio.
• Admit ordered chunks through the bounded native queue and acknowledge only after admission. Queue pressure must wait or cancel, never drop.
• Allow synthesis to run ahead only inside the bounded PCM horizon of the active playback backend.
• Treat LLM completion, producer finish, synthesis completion, resampler flush, and rendered drain as distinct milestones.
• Emit exactly one terminal speech outcome: drained, cancelled, or error.
• Cancel by speechSessionId and requestId; prevent old playback epochs or callbacks from changing replacement state.
• Keep Stop speaking speech-only for Chat. Preserve Live barge-in, mute, and End Live cancellation across generation, synthesis, queued PCM, and playback.
• Derive speaking and drained state from renderer evidence, not queue admission or buffer consumption.

Make performance changes experimentally

1. State the hypothesis and the metric it should improve: first audio, steady-state synthesis, RTF, gaps, CPU, RSS, or cancellation latency.
2. Run the release benchmark through just tts-benchmark; use exact verified production artifacts and fail instead of downloading or substituting.
3. Compare adapters independently. Include the unchanged candidate and reverse close contender order.
4. Inspect first-audio and total wall time together with audio duration, RTF, ordering, gaps, underruns, and cancellation.
5. Promote the smallest change that survives repeated and reversed runs. Keep current defaults when evidence is noisy or device acceptance is missing.
6. Document new evidence in AGENTS.md, benchmarks/tts/README.md, and docs/model-integrations.md when it changes the supported operating model.

Never call a configuration optimal from one fixture, one run, or one contender order. A faster RTF with unstable generated duration or worse wall time is not a production win.

Validate in layers

Run focused checks first, then broaden with the affected surface:

bun test
cargo test --manifest-path src-tauri/Cargo.toml --lib tts::stream
cargo test --manifest-path src-tauri/Cargo.toml --features tts-benchmark --bin tts-benchmark
just check

Run just bundle after Rust, Tauri, audio, command, or packaged UI changes. Invoke $test-voice-layer-app for packaged Chat/Live status, Stop speaking, Escape, focus, persistence, and real device acceptance. Distinguish rendered-drain evidence from audible quality; microphone, speaker/headphone continuity, echo cancellation, and barge-in require actual device exercise.

Completion bar

Finish only when:

• Chat and spoken Live still share ordering and lifecycle semantics without sharing the wrong playback graph;
• typed Live and submitted user prompts remain silent;
• first-chunk latency, ordering, backpressure, cancellation, and exactly-once drain remain covered;
• both adapters are evaluated independently when shared code changes;
• ordinary validation remains offline and does not fetch models;
• performance claims name the fixture, adapter, build mode, repeated-run method, and hardware boundary;
• AGENTS.md reflects any new durable TTS architecture or tuning decision.