converse

# converse - Real-Time Two-Way Voice Conversation Full-duplex voice conversation engine for personas. Not chatbot request-response — real conversation with interruptions, backchannels, emotional state, humming, and natural idle behaviors. ## Architecture ``` ConversationOrchestrator ├── Listener (VAD + Whisper STT, barge-in detection) ├── Thinker (EmbrySPARTAIntern + memory + mood + steering) ├── Speaker (PersonaPlex TTS with register switching) ├── Idler (humming, self-talk, backchannels, sighs) ├── Emotion (state machine with bridge attribute modifiers) ├── Mixer (audio ducking: speech > humming > background) └── Surface (Discord / KDE PipeWire / Horus WebRTC) ``` ## Quick Start ```bash cd .pi/skills/converse # Check everything is wired up ./run.sh sanity # Start conversation on KDE desktop ./run.sh start --surface kde --persona embry # With scenario attached ./run.sh start --surface kde --persona embry \ --scenario ../surf-qml/scenarios/design_review.yaml # Recorded mode (Qwen3-TTS fallback, not live PersonaPlex) ./run.sh start --surface kde --mode recorded # Discord voice ./run.sh start --surface discord ``` ## State Machine ``` IDLE → (user speaks) → LISTENING → (VAD end-of-turn) → THINKING THINKING → (response ready) → SPEAKING → (utterance complete) → IDLE SPEAKING → (barge-in: >300ms user speech) → INTERRUPTED → LISTENING IDLE → (>5s silence) → idler kicks in (humming, self-talk, backchannels) ``` ## Humming: Federated Taxonomy Graph Traversal Song selection for humming is not random. It uses multihop graph traversal: 1. **Current emotion** → active bridge attributes (Precision, Fragility, Resilience, etc.) 2. **Bridge attributes** → query episodic memory (`/episodic-archiver`) for resonant past experiences 3. **Bridge attributes** → query dream motifs (`/create-movie` → `horus-dreams` scope) for subconscious associations 4. **All of the above** → filter `/consume-music` registry for matching tracks per persona preferences 5. **Score candidates** by bridge overlap + episodic resonance + dream resonance 6. **Respect forbidden list** (persona-specific trauma triggers, e.g., Kamakawiwoole for Embry) Two-layer humming approach: - **PersonaPlex native**: Voice prompt conditioning with `[soft hum]`, `[mm]` tokens (Moshi fine-tuning) - **External music mixing**: Pre-generated fragments via `/create-music` MusicGen → RVC inference with Embry's voice model ## Emotion State Machine Tracks persona emotional state with bridge attribute modifiers: | State | Bridge Effect | |-------|--------------| | CURIOUS | Precision +0.1, Stealth -0.1 | | FRUSTRATED | Fragility +0.15, Corruption +0.05, Resilience -0.1 | | MELANCHOLIC | Fragility +0.2, Stealth +0.1, Corruption +0.05 | | OVERWHELMED | Fragility +0.25, Precision -0.15, Resilience -0.15 | Avoidance triggers (Embry-specific): - **Hard**: james, december 2023, ku'uipo, kamakawiwoole → deflection + anxious state - **Soft**: cabernet, steak salad, new haven winter → register shift to uncertain ## Components ### Listener (`listener.py`) - WebRTC VAD (aggressiveness=2) with energy-based fallback - Whisper large-v3 STT via faster-whisper or CLI - Barge-in: >300ms speech during playback → interrupt persona ### Thinker (`thinker.py`) - EmbrySPARTAIntern for grounded domain responses - Memory recall (persona + ToM scopes) - Persona journal mood context - Conversation steering presets - Avoidance trigger deflection - Interruption acknowledgment weaving ### Speaker (`speaker.py`) - PersonaPlex streaming TTS (live mode) - Qwen3-TTS / espeak-ng fallback (recorded mode) - Register-aware voice anchor selection (.pt files) - Sentence-boundary splitting for streaming ### Idler (`idler.py`) - Weighted behavior pool: backchannel (30%), self-talk (20%), hum (15%), sigh (10%), ambient (5%), silence (20%) - Cooldown per behavior to avoid repetition - Music selection via MusicSelector (Federated Taxonomy graph traversal) ### Mixer (`mixer.py`) - Named channels: speech, humming, background - Volume ducking with fade transitions - Rules: speech ducks humming to 0%, background to 10%; barge-in ducks everything ### Surfaces (`surfaces.py`) - Discord: Opus 48kHz via discord_voice_client - KDE: PipeWire pw-record/pw-play at 48kHz - Horus: WebSocket to gateway voice endpoint ## Session Storage All session data persists to `/mnt/storage12tb/converse-sessions/<persona>/<session_id>/`: - `transcript.jsonl` — timestamped turns with taxonomy tags - `emotion_log.json` — emotion state transitions over time - `session_summary.json` — duration, turn count, mood arc ## Dependencies ### Required - Python 3.10+, numpy, loguru - PipeWire tools (pw-record, pw-play) for KDE surface ### Optional (graceful degradation) - webrtcvad — falls back to energy-based VAD - faster-whisper — falls back to whisper CLI - PersonaPlex — falls back to Qwen3-TTS → espeak-ng - EmbrySPARTAIntern — falls back to Ollama llama3.1:8b - websockets — needed for Horus surface only ### Skill Dependencies - `/memory` — EmbrySPARTAIntern, persona_journal, MemoryClient - `/common` — memory_client - `/surf-qml` — taxonomy extraction pattern - `/train-convo-steering` — steering presets - `/consume-music` — music registry for humming - `/episodic-archiver` — past experiences for song selection - `/create-movie` — dream motifs for subconscious music associations - `/create-music` — RVC inference for Embry-voice humming - `/create-persona` — PersonaPlex config, detect_register() ## Preparation Steps (Before First Run) 1. **Train Embry humming voice model** via `/learn-voice` 2. **Pre-generate melodic fragments** via `/create-music` MusicGen → RVC 3. **Create humming voice prompt (.pt)** for PersonaPlex native backchannels 4. **Fine-tune Moshi LoRA** with labeled humming data (`[humming]`, `[mm]`, `[soft hum]` tokens) ## Troubleshooting Run `./run.sh sanity` to check all dependencies and assets. Common issues: - **No audio**: Check PipeWire is running (`pw-cli info`) - **VAD too sensitive**: Increase aggressiveness in listener.py (0-3) - **Slow transcription**: Ensure CUDA available for faster-whisper - **No humming**: Check consume-music registry has tracks and hum-cache exists