esp-hermes-channel

# ESP-Hermes Channel Turn an ESP32-S3 (M5Stack) into a **Hermes voice channel** — parity with Telegram/Desktop clients, NOT a standalone agent. Hermes (gateway) does STT + reasoning + tools + TTS + pet-state; the ESP does I/O only. ## Architecture ``` ESP32-S3 (esp-hermes firmware, C/Arduino) mic ──audio──▶ Hermes Gateway (:9119, like Telegram channel) │ ├─ STT (local faster-whisper / groq) ├─ Agent reasoning + tools + memory ├─ TTS (Edge TTS) ──audio──▶ speaker └─ Pet-State push (idle/run/review/error/done) ──▶ LCD ``` - ESP is a **gateway client**, not an LLM client. Do NOT use ESP-Claw's LLM mode — that makes the ESP a standalone agent. We want Hermes as the brain. - Channel parity table: | Feature | Desktop | Telegram | ESP32 | |---|---|---|---| | Send text/audio | ✓ | ✓ | ✓ (button→mic) | | Agent replies | ✓ | ✓ | ✓ (TTS→speaker) | | Pet on display | ✓ | ✗ | ✓ (LCD) | | Tool-calls | ✓ | ✓ | ✓ (via gateway) | ## Full Hardware Access (Hermes-controlled IoT Node) ESP is a **gateway client**, so the agent gets full tool-call access to the S3's IO — the device becomes Hermes' physical extremity. | IO | Access via Hermes | Example | |---|---|---| | **GPIO** (digital in/out) | Tool-Call → ESP sets/reads pin | relay, LED | | **I2C / SPI** (sensors, displays) | Tool-Call → Bus-Read/Write | external sensors | | **ADC** (analog, voltage, temp) | Tool-Call → `analogRead` | battery, thermistor | | **PWM** (LEDs, servos, motors) | Tool-Call → `ledcWrite` | dimming, servo | | **UART** (other MCUs) | Tool-Call → Serial passthrough | chain devices | | **Button/Touch** | Input-Event → triggers Hermes | PTT, mode toggle | | **IMU — Accelerometer + Gyroscope** | Tool-Call `esp_imu_read` **+ Event-Driven triggers** | motion/orientation/gesture | | **LCD** | Pet-State **+** arbitrary render output | pet + status | | **mic/speaker** | Voice-Channel (as discussed) | talk to Hermes | ### IMU (Accelerometer + Gyroscope) — special case M5Stack S3 includes a 6-axis IMU (accel + gyro). Not just readable — **event-driven**: - **Polled:** `esp_imu_read` → returns accel/gyro vector on demand ("what's the orientation?"). - **Event-Driven (recommended):** on-device motion detection triggers Hermes events without a prompt: - **Shake** → toggle always-on / PTT mode - **Tap** → PTT trigger (alt to button) - **Orientation flip** → context switch ("device picked up" → wake) - **Stillness timeout** → return to `idle` pet state - ESP runs lightweight motion filter (threshold/low-pass) so it doesn't spam the gateway with noise. Mirrors ESP-Claw's "Event Driven" feature — but the *brain* stays Hermes (gateway), not on-device Lua. ### Design Principles (user-driven — learned this session) - **Device = Hermes channel, not standalone agent.** ESP speaks to the gateway (WS); Hermes is the brain. Do NOT use ESP-Claw's LLM mode. - **Reuse vendor SDK/components** (ESP-Claw `common`/`edge_agent`) instead of rewriting from scratch. Strip the LLM client, keep hardware/IMU/event layers. - **Start small, phased** (voice-pet first, body later). Ship Phase 1, grow incrementally. User explicitly: "wir fangen klein an". - **MCP direction caution:** a device's MCP adapter usually makes the *device* the MCP *client* (it calls servers), so it does NOT grant Hermes access to the device. Don't assume reverse direction. - **Verify spec/benchmark numbers before stating them.** User corrects confident wrong claims — e.g. 200 TOPS NPU ≠ faster LLM than his S24 (real token speed wins); and LMArena ELO for gemma-4-31b (1451) was wrongly attributed to the 2B mobile variant. Check the number matches the exact model/size before claiming. ## Two Modes (REQUIRED) 1. **Push-to-talk (PTT):** Hold button → record → release → upload. Pet shows `recording` state while held. 2. **Always-on (VAD):** Continuous mic, on-device Voice Activity Detection triggers upload only on speech. Pet shows `listening` idle state; `run` while Hermes processes. Mode toggle: button short-press cycles modes, or gateway config push. Persist mode in ESP NVS. ## Audio Pipeline - Upload: raw PCM or OGG/Opus (M5 mic → encode → POST to gateway `/voice` or webhook). - Downlink: TTS audio streamed back → ESP decodes → speaker. - Codec decision pending hardware arrival (I2S mic + MAX98357 or built-in M5 speaker). ## Pet on LCD - petdex renderer runs on the **gateway** (Hermes side), not ESP. - Gateway pushes `pet_state` + frame data over the same channel socket (WebSocket recommended). - ESP LCD renders frames for current state. States: `idle`, `run`, `review`, `error`, `done`. - Pet slug chosen by user (same gallery as desktop — `hermes pets list`). ### petdex scope vs. custom IMU layer (IMPORTANT) petdex (verified): 3,317+ pets, Codex sprite format, 192×208 grid. Pets react to **agent-lifecycle states** (`idle`/`run`/`review`/`error`/`done`) — this is the foundation and works out-of-the-box for the ESP LCD. **Gap:** petdex does NOT react to physical device motion. IMU coupling (pet leans on tilt, dizzy on shake) is NOT a native petdex feature. Two integration paths: - **Path A (safe, compatible):** IMU → gateway maps motion to existing petdex states (`run`=busy, `done`=content). Always compatible, no custom frames. - **Path B (rich, custom):** add custom pet states (`tilt`, `shake`, `stretch`) — requires pets whose atlases contain those frames. Needs custom pet or petdex atlas extension. **Decision:** petdex = foundation (agent states). Physical IMU coupling = custom layer on top (Path A for v1, Path B later). The gateway controls the state-push, so we own the mapping. ### Video on state change (optional, theoretical) Beyond sprite pets, the LCD can play **short video clips** on state transitions: - Gateway pushes `play_video: <clip-id>` instead of (or alongside) `pet_state`. - Feasible formats on ESP32-S3: MJPEG (TJpgDec per-frame), animated GIF, or raw RGB frames. Avoid MP4/H.264 (no hardware decoder on S3). - Resolution cap: ∼QVGA (320×240) or the Stick-S3 native panel. Keep clips 1–3s. - **Best mix:** sprite pet as persistent idle mode (cheap, smooth); video burst only on event (`error`→glitch clip, `done`→celebrate clip). - Storage: clips on SD/SPIFFS or streamed from gateway over WiFi. ## Transport / Connectivity - ESP connects to Hermes gateway over WiFi (HTTP or WebSocket). - Gateway must be reachable: Cloudflare tunnel or VPN (same as S24 setup). - Auth: API key / basic auth in front of endpoint. - Recommended: WebSocket for bidirectional (audio up + state/TTS down) to avoid polling. ## IO-Tool Layer (Channel Registration) On connect, the ESP **registers its IO tools** with the gateway so Hermes sees them like any other tool. Define a stable schema: ```json { "tool": "esp_gpio_set", "device": "<esp-id>", "params": { "pin": 12, "state": "HIGH" } } { "tool": "esp_gpio_read", "params": { "pin": 14 } } { "tool": "esp_adc_read", "params": { "pin": 36, "atten": 11 } } { "tool": "esp_pwm_set", "params": { "pin": 13, "duty": 2048, "freq": 5000 } } { "tool": "esp_i2c_read", "params": { "addr": 64, "reg": 0, "len": 2 } } { "tool": "esp_i2c_write", "params": { "addr": 64, "reg": 0, "data": [1,2] } } { "tool": "esp_uart_send", "params": { "data": "AT+CMD\r\n" } } { "tool": "esp_imu_read", "params": { "axis": "all" } } ``` - Hermes routes these as normal tool-calls; the gateway serializes → WebSocket → ESP → Arduino handler. - ESP replies with JSON result (value / ack / error). - **Dynamic discovery:** ESP sends a `capabilities` message on connect so the gateway knows which pins/peripherals exist (avoids hardcoding per-device). ## Safety (CRITICAL — full HW access = physical consequences) Because Hermes can move motors, flip relays, fire PWM, the channel needs guardrails: 1. **Pin Allowlist** — `config.yaml` under `esp_hermes.allowed`: ```yaml esp_hermes: devices: stick-s3: allowed_pins: [12, 13, 14, 36] # only these are callable allowed_i2c: [0x40, 0x68] # sensor addresses only blocked_pins: [0, 1, 2, 3] # boot/flash/UART0 never exposed ``` 2. **Destructive-action approval** — mirror `approvals.mode`. Critical IO (relay, motor PWM, any `gpio_set` on allowlisted power pin) prompts the user before execution unless `esp_hermes.auto_approve_safe: true`. 3. **Rate-limit** — cap `esp_pwm_set` / `esp_gpio_set` calls per second to prevent strobe / burn-out loops. 4. **Power pins protected** — never expose EN, MTDI, GPIO0/1/2/3 (boot/flash). Hard-block in firmware regardless of allowlist. 5. **IMU event debounce** — on-device low-pass + threshold filter; only emit `tap`/`shake` after N stable samples (no gateway spam). 6. **Audit log** — every IO tool-call logged to `~/.hermes/logs/esp_hermes.log` with timestamp + device + pin + value. ## Physical Interaction Beyond Wake (Embodied Hermes) The device is not just an input surface — **movement in 3D space is meaningful**. Hermes gets a "body" via the IMU + motor control. Start small, expand later. ### Pet Animation ↔ Motion Coupling The pet on the LCD must **react to physical movement**, not just agent state: | Physical action | Pet reaction | Optional task trigger | |---|---|---| | Lift / pick up | pet "wakes up", stretches | none (ambient) | | Shake | pet gets dizzy / spins | toggle mode | | Tilt left/right | pet leans, looks curious | none | | Set down gently | pet settles to `idle` | none | | Fast spin | pet excited 🌀 | easter egg | | Flip upside down | pet confused 🙃 | none | → IMU orientation vector drives pet **pose/animation**, not just events. Smooth interpolation (not snapping) so the pet "feels" alive. ### Movement-Triggered Tasks Beyond wake: specific gestures = specific Hermes actions (user-definable mapping): - **Double-tap + tilt** → start a named cron/ritual - **Figure-8 motion** → "summarize my day" task - **Knock on table (IMU shock)** → quick status ping These are event-driven tool-calls, same mechanism as mode-toggle, but **mapped to user tasks** via config. ### Easter Eggs Motion-based hidden behaviors (fun + delight): - **Secret shake pattern** → pet does a dance, plays a sound - **Upside-down hold 3s** → pet "sleeps", LCD dims - **Rapid 3x tap** → surprise animation Keep them on-device (cheap, no gateway) so they feel instant. ### Motor Control (Hermes gets a body) Later phase: attach a motor/servo/RGB — Hermes can **move** the device or its surroundings: - `esp_motor_set` → rotate / nudge / point - Physical "nod" on task complete, "shake" on error (mirrors pet states in 3D) - Turns the channel from a passive speaker into an **embodied agent** ### Phased Rollout (start small) 1. **Phase 1 (first boot):** PTT + always-on voice + pet on LCD (idle/run/done). 2. **Phase 2:** IMU wake + tap/shake mode-toggle + pet motion-coupling (lean/dizzy). 3. **Phase 3:** IO-tool layer (GPIO/I2C/PWM) + safety allowlist. 4. **Phase 4:** movement-triggered tasks + easter eggs. 5. **Phase 5:** motor control — Hermes has a body. Never block Phase 1 on later phases. Ship the voice-pet first, grow the body incrementally. ## Open Questions (resolve at hardware arrival) - [ ] Audio codec (Opus vs PCM) — bandwidth vs ESP decode cost - [ ] State-push protocol (WS msg schema) — define with gateway team - [ ] Pet frame format for LCD (sixel? custom sprite? downscaled 192×208) - [ ] Power: always-on VAD battery impact - [ ] Wake-word for always-on mode (optional, on-device) ## Build Checklist (when hardware arrives) - [ ] Flash esp-hermes firmware (C/Arduino, WiFi + I2S + LCD) - [ ] Point ESP at gateway WebSocket URL + auth - [ ] Verify PTT: button→mic→STT→TTS→speaker - [ ] Verify always-on: VAD trigger - [ ] Wire pet-state push → LCD render - [ ] Select pet slug via `hermes pets` - [ ] Tunnel/gateway auth hardened ## Development Workflow (proven this session) How the project was actually scaffolded — reuse for future builds: 1. **Kanban board** `esp-hermes` (Hermes `kanban`) — 7 cards map to spec sections (gateway-adapter / ws-hub / io-tools / config-safety / commands / firmware-draft / repo-scaffold). Each card tagged with `esp-hermes-channel` skill so JIT workers load context. 2. **JIT agents on Modal** build the gateway-side code (Python) in parallel — firmware is **draft-only** until hardware arrives (no flash/test possible without the device). 3. **Repo layout:** `ohrbit/hermes_plugins` → subfolder `esp-hermes/` (NOT hermes_skills). Structure: `gateway/`, `tools/`, `firmware/`, `config/`, `references/`, hand-authored `README.md` (use `github-readme-authoring` skill — no generators), `LICENSE` (MIT), `implementation-spec.md`. 4. **GitHub push without `gh`:** `gh` CLI is NOT installed in this env. Use `GITHUB_TOKEN` from `~/.hermes/.env` + curl API + `git remote set-url origin https://$GITHUB_TOKEN@github.com/<repo>.git`. Verify push via `GET /repos/<repo>/contents/<path>` (expect 200). 5. **Tunnel-Derby lesson applies:** when dispatching JIT workers, pass `skills=[esp-hermes-channel]` and confirm Modal RAM before fan-out. ### Kanban / JIT dispatch — exact commands that worked (learned the hard way) - **Board + cards:** `hermes kanban --board <name> create "Title" --body "..." --skill esp-hermes-channel` (the `--board` flag goes BEFORE the subcommand; `--title`/`--board` AFTER it error out).