embry-voice-control

# Embry Voice Control Use this skill when an agent needs to operate Embry voice as a controllable system. This is the endpoint/control-plane contract. It is not the conversation style guide and it is not the Chatterbox renderer itself. Think of `embry-voice-control` as the voice front-end to Tau: it receives voice or text turns, gathers listener and memory evidence, asks Tau to shape the agentic response when needed, and then sends approved render text to Chatterbox. It must not become a second reasoner beside Tau. ## Boundary - `$memory` owns speaker identity, intent, recall, answer, clarify, and deflect. - Tau owns agentic reasoning and project/tool coordination; this skill fronts Tau with voice/chat control. - Chatterbox owns speech rendering from approved `tts_render_text`. - RealtimeSTT/listener owns VAD, ASR, diarization, and speaker evidence. - Shared Chat UX owns visible transcript, reasoning trace, audio controls, orb state, and replay presentation. - `embry-voice-control` owns the agent-readable endpoint contract connecting those systems for one turn/session. ## Required Endpoint Surface Implementations should expose JSON endpoints with stable request/response schemas: ```text GET /health GET /state GET /readiness POST /listen/start POST /listen/stop POST /turn POST /speak POST /turn/cancel POST /replay GET /sessions GET /sessions/{session_id} GET /turns/{turn_id} GET /receipts/{turn_id} ``` The endpoint names are the control contract. A current project may temporarily adapt them to local URLs such as `/api/projects/embry-voice/live-turn`, but agents should treat this skill as the stable target shape. ## Turn Flow ```text voice/text turn -> /turn -> listener evidence when voice is enabled -> $memory /speaker/resolve when speaker evidence exists -> $memory /intent, including voice delivery policy -> $memory /recall and /answer | /clarify | /deflect as routed -> Tau response shaping when needed -> Chatterbox /tau/voice-render or /synthesize-batch-stream -> shared Chat UX transcript + audio + orb + replay receipt ``` Every response must preserve a single turn authority: ```text turn_id == chat receipt turn_id == audio artifact authority == orb source ``` ## Speaking `POST /speak` is for direct approved speech. It must not invent factual answers. Required request fields: - `turn_id` - `session_id` - `tts_render_text` - `answer_text`, when different from rendered text - `tone` - `delivery_stage` - `pause_strategy` - `interrupt_policy` - `interruptible` - `play_local`, when speaker playback is requested Required response fields: - `mocked` - `live` - `turn_id` - `conversation_tone` or `tone` - `delivery_stage` - `emotion_tags` - `chatterbox_tags` - `cue_policy` - `intent_policy_source` - `pause_strategy` - `interrupt_policy` - `audio_artifact_id` - `audio_url` or `audio_path` - `audio_authority` - `voice_envelope`, when orb sync is supported - `receipt_path` ## Listening `POST /listen/start` and `POST /listen/stop` control listener state only. They do not directly answer the user. The authoritative listener path is Unix/PipeWire audio capture feeding RealtimeSTT from a local service or runner. Browser microphone/WebRTC capture is optional client input only. It must not be the primary acceptance path because Chrome site permissions, tab activation, and device binding have proven brittle for `localhost:3002`. The shared Chat UX should subscribe to listener state and transcript events from the Unix listener service. React should not own wake-word recognition. A browser mic button may request permission and forward audio when it works, but a failed browser permission prompt must not block the system-level listener. The required wake word is exactly: ```text Embry ``` The visible standby affordance should say `SAY "EMBRY"`. When the local listener detects the wake word, it must emit a wake event and transition: ```text idle -> wake_detected -> listening ``` The UI should dim the `SAY "EMBRY"` instruction and show `LISTENING...` as soon as the wake event is accepted. The listener should reject low-confidence, non-primary-speaker, cooldown, and while-Embry-is-speaking wake attempts. The listener service should emit events through an explicit local transport, preferably a Unix socket or localhost HTTP/SSE/WebSocket adapter: - `listener.state`: idle, listening, transcribing, processing, error - `listener.wake_detected` - `listener.partial_transcript` - `listener.final_transcript` - `listener.error` - `listener.receipt_written` Listener output must become structured turn evidence: - `audio_turn_started` - `partial_transcript` - `final_transcript` - `speaker_candidates` - `diarization_segments` - `overlap_detected` - `interruption_detected` - `barge_in_detected` Speaker evidence must be passed to `$memory /speaker/resolve`; the voice control layer must not choose identity by label assumption. ## Replay `POST /replay` should replay the conversation through the same Chat UX timeline, including user turns, Embry responses, Chatterbox audio, reasoning trace, and orb state. Replay is not only audio playback; it is a reconstruction of the turn sequence with timing offsets. ## Required Behavior - Voice input is as first-class as text input. - `$memory /intent` is the authority for Embry's conversational tone and injected emotion/tag policy on normal generated turns. The voice control layer must not guess tone or Chatterbox tags locally when memory intent is available. - Every Embry speech item must include a selected conversational tone and an injected-emotion/tag policy. If no literal Chatterbox tag is appropriate, the receipt must explicitly record `chatterbox_tags=[]` and explain the `cue_policy`; omission is a failed receipt. Receipts must also record `intent_policy_source`, normally `memory.intent`; direct local sanity speech may use `direct_sanity_explicit_policy`. - Every Embry speech item must include the default pause and interrupt strategy. `pause_strategy` and `interrupt_policy` are required receipt fields, not UI hints. Missing pause or interrupt policy is a failed voice receipt. - Unknown or ambiguous speakers fail closed to identity clarification. - Multiple non-Embry speakers overlapping map to a one-at-a-time boundary. - Barge-in cancels old speech and stale chunks before the new turn wins. - Chat text, audio, orb, and receipts must share turn authority. - Any cached QRA audio path must be gated by `$memory` near-exact approved QRA recall for the resolved speaker. - All claims of working behavior require non-mocked receipts that state `mocked`, `live`, what was exercised, and what remains unverified. ## Implementation Standards If this skill grows scripts or a service, follow `$best-practices-python`: Loguru, Typer, httpx, uv/pyproject, module docstrings, thin `__init__.py`, complete dependencies, functions first, files under 800 lines, and non-mocked sanity tests. ## Live Sanity Run the deterministic wake-word contract sanity before live listener work: ```bash ./run.sh wake-sanity ``` This writes a receipt under `/mnt/storage12tb/skills/embry-voice-control/outputs/e2e/wake-word/`. It proves only the local wake-word state contract and does not prove hot mic capture, RealtimeSTT wake detection, Chatterbox, Chat UX, or orb sync. Run the Unix/PipeWire RealtimeSTT listener rung before claiming live listener readiness: ```bash ./run.sh unix-listener-sanity ``` This wraps the local RealtimeSTT PipeWire proof runner and writes receipts under `/mnt/storage12tb/skills/embry-voice-control/outputs/e2e/unix-listener/`. Pass requires real PipeWire playback/capture, non-silent captured audio, RealtimeSTT realtime and final transcript callbacks, final transcript match, and wake word detection from the transcript. It does not prove browser mic, speaker identity, Tau/memory, Chatterbox, Chat UX, orb, replay, or interruption. Run the persistent local event journal before continuous listener proofs: ```bash ./run.sh listener-service \ --host 127.0.0.1 \ --port 8019 \ --db-path /mnt/storage12tb/skills/embry-voice-control/state/voice-events.sqlite3 ``` RealtimeSTT proof runners publish live callbacks to `POST /v1/listener/events`. The service persists canonical `embry.voice_event.v1` rows in SQLite WAL and exposes ordered session journals at `GET /v1/sessions/{session_id}/journal`. Event IDs are idempotent, while conflicting IDs or session sequence numbers fail closed. This replaces `latest_unix_listener.json` as the event authority; that compatibility pointer must not be used for suite-readiness audits. Run the listener-to-turn rung after `unix-listener-sanity`: ```bash ./run.sh listener-turn-live ``` This consumes the latest Unix listener receipt, strips the leading `Embry` wake word from the final transcript, sends the voice-mode turn to `/live-turn`, and requires memory/Tau evidence plus Chatterbox audio authority. The default semantic check expects the answer to mention `Paris` for the current capital-of-France sanity phrase. A failed semantic answer is a failed rung even when Chatterbox renders fallback audio. To make the Chatterbox artifact audible during this rung, run: ```bash ./run.sh listener-turn-live --play-local --local-playback-target 64 ``` The receipt must record the `pw-play` command and whether local PipeWire playback returned successfully. Audible playback does not override semantic answer failure. Run the Embry Chat Tau-style static query rung before trusting the legacy UX Lab `/live-turn` adapter as the reasoning path: ```bash ./run.sh embry-chat-static-query-live --play-local --local-playback-target 64 ``` This consumes the latest Unix listener receipt and proves the narrow voice front-end contract: ```text RealtimeSTT final transcript -> Embry Chat Tau subagent boundary -> $memory /intent first -> $memory /answer second -> irrelevant or deterministic memory answer is classified as a miss -> Tau static-answer route for "What is the capital of France?" -> Chatterbox receives only Tau-approved tts_render_text -> optional PipeWire playback receipt ``` The receipt schema is `embry_chat_turn_receipt.v1`. Required evidence includes `intent_call_index=1`, `answer_call_index=2`, `route_taken=static_answer`, `tts_render_text` matching the Tau response plan, a readable Chatterbox WAV, and `pw-play` return code `0` when `--play-local` is used. This rung does not prove browser mic, speaker identity, Chat UX rendering, orb sync, replay, interruption, or the 200+ stress suite. The first operational Tau DAG wrapper for this rung lives at: ```text proofs/tau/embry-chat-static-query-live/dag-contract.json ``` It declares the persistent Embry voice surface in `nodes[].persistent_subagent`, uses `embry-chatterbox-voice` as the bounded persistent surface node, calls this skill through a Tau command spec, and emits `tau.agent_handoff.v1` with `persistent_subagent_receipt`, `embry_chat_turn_receipt.v1`, `chatterbox_audio_receipt`, and `pipewire_playback_receipt`. Run the research routing rung before claiming Embry can answer current-world non-compliance questions through the voice front-end: ```bash ./run.sh research-turn-live ``` This first calls `$memory /intent` for an explicit research query and requires `action=RESEARCH` plus `brave-search` in the intent tooling. It then sends the same query to the configured Embry `/live-turn` adapter and fails unless the live response records a non-mocked `research.brave_search` call with web results and Chatterbox audio authority. It does not prove browser mic, RealtimeSTT input, speaker identity, Chat UX rendering, orb sync, replay, or interruption. Run the live wake/control-plane runner when the UI already exists but Embry does not wake into a turn yet: ```bash ./run.sh wake-capital-france-live --base-url http://127.0.0.1:3001/api/projects/embry-voice ``` This writes a receipt under `/mnt/storage12tb/skills/embry-voice-control/outputs/e2e/wake-capital-france/`. It proves a controlled wake event is produced and fed to the live `/live-turn` control endpoint with the spoken-style question "What is the capital of France?". It must fail if the endpoint does not return live non-mocked turn/audio authority or if the answer does not mention Paris. It does not prove hot microphone wake detection or browser WebRTC capture. Run the opt-in live harness before claiming the voice front-end works: ```bash ./run.sh verify --profile controlled-live ``` The harness calls real configured endpoints and writes receipts under `/mnt/storage12tb/skills/embry-voice-control/outputs/e2e/<run-id>/`. Missing services, missing fields, stale turn authority, absent Tau evidence, or absent audio/orb authority must produce `NOT_ESTABLISHED` or `NOT_READY`, not a mocked pass. ## References - `references/endpoint-contract.md`: request/response shapes and misuse cases. - `references/e2e-sanity-checks.md`: live non-mocked sanity matrix.