realtime-voice-2

name	realtime-voice-2
description	Add OpenAI Realtime API voice capabilities (GPT-Realtime-2 reasoning voice agent, GPT-Realtime-Translate live speech translation, GPT-Realtime-Whisper streaming transcription) to Hermes Agent. Use this skill whenever the user wants live voice conversation with Hermes, real-time translation, streaming captions, or wants to wire OpenAI Realtime API into the Hermes gateway. Triggers on phrases like "talk to Hermes", "voice mode", "realtime voice", "GPT-Realtime-2", "live translation", "streaming transcription", "WebRTC voice", or "OpenAI voice agent". Works alongside Hermes's existing voice memo transcription — this adds bidirectional speech-to-speech rather than turn-based memo handling.
version	1.0.0
author	arananet
metadata	{"hermes":{"tags":["voice","openai","realtime","translation","transcription","webrtc"],"category":"media","requires_tools":["terminal"],"config":[{"key":"realtime_voice_2.default_voice","description":"Default GPT-Realtime-2 voice (cedar or marin)","default":"marin","prompt":"Default voice for the realtime agent"},{"key":"realtime_voice_2.default_target_language","description":"Default ISO 639-1 code for translation output","default":"es","prompt":"Default target language code for translation"},{"key":"realtime_voice_2.reasoning_effort","description":"Reasoning effort for gpt-realtime-2 (minimal|low|medium|high|xhigh)","default":"low","prompt":"Reasoning effort level"},{"key":"realtime_voice_2.server_port","description":"Local port for the WebRTC bridge server","default":3737,"prompt":"Server port"}]}}
required_environment_variables	[{"name":"OPENAI_API_KEY","prompt":"OpenAI API key (sk-proj-... or sk-...)","help":"Get a key from https://platform.openai.com/api-keys. Must have Realtime API access.","required_for":"full functionality"}]

Realtime Voice 2

Bring OpenAI's Realtime API into Hermes. Three modes, one skill:

1. Voice agent (gpt-realtime-2) — speech-to-speech with Hermes's personality, tool calls, 128K context, GPT-5-class reasoning
2. Live translation (gpt-realtime-translate) — 70+ input languages → 13 output languages, streamed
3. Streaming transcription (gpt-realtime-whisper) — live captions with controllable latency

Hermes already does voice-memo transcription. This skill is different — it adds bidirectional, low-latency voice. The two coexist: voice memos for turn-based messaging platforms (WhatsApp, Telegram); this skill for live conversation.

---

When to Use

User says	Mode	Slash
"talk to me", "voice mode", "start a call"	Voice agent	/realtime-voice-2 talk
"translate this stream to Spanish", "be my interpreter"	Translation	/realtime-voice-2 translate <lang>
"transcribe what I'm saying", "live captions"	Transcription	/realtime-voice-2 transcribe
"stop voice", "end call"	Tear down	/realtime-voice-2 stop

If the user is on a messaging gateway (Telegram, WhatsApp, etc.) and asks for voice, point them at the CLI or the web UI — WebRTC needs a browser or native audio I/O, which messaging platforms don't provide directly.

---

Procedure

1. Verify environment

Before doing anything else, check that OPENAI_API_KEY is set (Hermes will have prompted for it on skill load if missing). Verify the key actually works against the Realtime API:

curl -sS https://api.openai.com/v1/models/gpt-realtime-2 \
  -H "Authorization: Bearer $OPENAI_API_KEY" | jq .id

If the key lacks Realtime API access, the user gets a 404 or 403. Tell them to enable Realtime on their OpenAI org.

2. Decide which mode

Ask the user only if it's ambiguous. Most requests are clear:

• "talk to me" → voice agent
• "translate" → translation
• "captions" / "transcribe" → transcription

For voice agent, also ask if they want Hermes's full personality (SOUL.md) or a stripped-down assistant. Default: full personality.

3. Start the local bridge

The bridge is a small Python server (FastAPI + aiohttp) that:

• Serves the web UI at http://localhost:<port>/
• Proxies SDP offers to OpenAI's /v1/realtime/calls with the API key
• Bridges function-tool calls back into Hermes's tool gateway

Run it from the skill scripts:

python "$HERMES_SKILLS/realtime-voice-2/scripts/bridge.py" \
  --mode voice \
  --voice "${realtime_voice_2.default_voice}" \
  --reasoning "${realtime_voice_2.reasoning_effort}" \
  --port "${realtime_voice_2.server_port}"

For translation mode, pass --mode translate --target-lang es. For transcription, --mode transcribe.

The script reads OPENAI_API_KEY from env (Hermes passes it through to sandboxes automatically — that's why we declared it in required_environment_variables).

4. Open the UI

Tell the user to open http://localhost:<port>/ and grant microphone access. The UI has three tabs corresponding to the three modes — the --mode flag pre-selects the right one.

5. Inject Hermes personality (voice agent mode only)

When the bridge starts in voice mode, it reads ~/.hermes/SOUL.md and the user's profile if available, and sends them as session.instructions over the data channel. This is what makes the voice agent sound like Hermes rather than a generic assistant.

soul_path = Path.home() / ".hermes" / "SOUL.md"
instructions = soul_path.read_text() if soul_path.exists() else ""
instructions += "\n\nYou are speaking to the user via live voice. " \
                "Keep responses concise. Use short preambles like " \
                "'let me check' before tool calls."

6. Wire tool calls back into Hermes

GPT-Realtime-2 can call functions. The bridge translates response.function_call_arguments.done events into Hermes tool dispatches via the local tool gateway HTTP API:

# When the voice model wants to call a tool:
result = await http_post(
    f"http://localhost:{HERMES_TOOL_GATEWAY_PORT}/dispatch",
    json={"tool": event["name"], "arguments": json.loads(event["arguments"])},
)
# Send result back to the model:
dc.send(json.dumps({
    "type": "conversation.item.create",
    "item": {
        "type": "function_call_output",
        "call_id": event["call_id"],
        "output": json.dumps(result),
    },
}))
dc.send(json.dumps({"type": "response.create"}))

This means the voice agent has access to every Hermes tool the current session has access to — terminal, web search, MCP servers, scheduled tasks, memory operations, the full toolset. The user can say "remind me tomorrow at 9am to call Mike" and it goes through the same cron tool the text gateway uses.

7. Persist the conversation

When the call ends, save the full transcript to Hermes's session history so it's searchable via FTS5 the same as any other conversation. The bridge emits a final JSON to stdout that Hermes captures.

print(json.dumps({
    "hermes_session_record": {
        "modality": "voice",
        "model": "gpt-realtime-2",
        "transcript": full_transcript,
        "duration_seconds": elapsed,
        "tool_calls": tool_call_log,
    }
}))

---

Pitfalls

These are the failures I've seen people hit. Read this before debugging.

400 Bad Request on /v1/realtime/calls

The classic. You appended ?model=... to the URL. Don't — the model goes inside the session JSON in the multipart body.

Multipart parts are corrupted

Don't hand-roll the multipart body. Use aiohttp.FormData() or Python's requests.post(..., files=...). Raw SDP contains \r\n and can collide with a fixed boundary string, splitting your form into garbage.

Voice agent uses someone else's personality

You forgot to pass session.instructions. By default the model is a generic assistant. Inject SOUL.md after the data channel opens.

Translation mode returns silence

Source language equals target language. The model is interpretation-only — it won't translate English to English. Add a UI hint or fall back to original-audio passthrough for those segments.

response.create breaks translation

Translation sessions don't use the assistant lifecycle. Never call response.create on a translation session. Translation runs continuously from the input audio stream itself.

Tool calls return but model doesn't speak

You sent function_call_output but forgot the follow-up response.create. The model needs both events to continue.

Sessions die at 30 minutes

Direct API key sessions have a 30-minute TTL. For longer calls, mint an ephemeral token (POST /v1/realtime/client_secrets) and use it from the browser — that gives 2 hours. The bridge supports both modes via --auth-mode ephemeral.

Mic permission denied on first try

Browsers require HTTPS for getUserMedia except on localhost. The bridge binds to localhost for this reason. If you proxy it through a tunnel, use HTTPS at the tunnel.

Hermes can't reach the tool gateway from the bridge

The bridge expects HERMES_TOOL_GATEWAY_PORT in env. Hermes sets this automatically when launching skill scripts. If you run the bridge outside Hermes (testing), set the port manually or pass --no-tools.

---

Verification

Before declaring the session working, confirm:

[ ] Bridge starts without errors, prints "listening on http://localhost:<port>"
[ ] Browser UI loads, microphone permission granted
[ ] Voice agent: model responds in spoken English within ~2-3s
[ ] Voice agent: speaks Hermes's personality (recognizable from SOUL.md)
[ ] Voice agent: interrupting mid-response actually interrupts
[ ] Voice agent: a Hermes tool call (e.g. "search the web for X") fires
[ ] Voice agent: transcript saved to Hermes session history after end
[ ] Translation: model speaks in target language, latency under 1s
[ ] Transcription: text deltas appear as you speak, not only after silence
[ ] Network tab shows no API key in any browser-served file

---

Mode reference

Voice agent — full session.update payload

{
  "type": "realtime",
  "model": "gpt-realtime-2",
  "audio": { "output": { "voice": "marin" } },
  "reasoning": { "effort": "low" },
  "instructions": "<SOUL.md contents + voice-specific addenda>",
  "tools": [
    {
      "type": "function",
      "name": "hermes_dispatch",
      "description": "Call any Hermes tool by name with arguments.",
      "parameters": {
        "type": "object",
        "properties": {
          "tool_name": { "type": "string" },
          "tool_arguments": { "type": "object" }
        },
        "required": ["tool_name", "tool_arguments"]
      }
    }
  ]
}

Why a single hermes_dispatch function rather than registering every tool? Hermes has 40+ tools and that's a lot of tokens in the session config. The dispatch wrapper keeps the session small and lets the model name any tool dynamically. If a specific tool would benefit from a typed wrapper (e.g. schedule_reminder(when, what)), register it explicitly in addition to the dispatcher.

Translation — session config

{
  "session": {
    "audio": {
      "input": {
        "transcription": { "model": "gpt-realtime-whisper" },
        "noise_reduction": { "type": "near_field" }
      },
      "output": { "language": "es" }
    }
  }
}

Endpoint: wss://api.openai.com/v1/realtime/translations?model=gpt-realtime-translate

Translation does NOT support tools, voice selection, or system prompts. The output voice adapts to the source speaker's tone.

Transcription — session config

{
  "type": "transcription_session.update",
  "session": {
    "input_audio_format": "pcm16",
    "input_audio_transcription": {
      "model": "gpt-realtime-whisper",
      "language": "en"
    },
    "turn_detection": {
      "type": "server_vad",
      "threshold": 0.5,
      "silence_duration_ms": 500
    }
  }
}

Lower silence_duration_ms for snappier deltas; raise it for cleaner final text. gpt-realtime-whisper exposes a controllable latency knob via the turn_detection config — tune on real audio, not synthetic clips.

---

Pricing reference (Nov 2025 GA)

Model	Pricing
gpt-realtime-2	$32 / 1M audio in tokens ($0.40 cached), $64 / 1M audio out tokens
gpt-realtime-translate	$0.034 / minute
gpt-realtime-whisper	$0.017 / minute

Translation and Whisper are billed by audio duration, not tokens. Don't use gpt-realtime-2 where Whisper or Translate is the right tool — Whisper is ~half a cent per minute.

---

Reference files

Loaded on demand via skill_view:

• references/webrtc-flow.md — SDP / multipart / data channel spec; ephemeral tokens
• references/translation-config.md — translation session details, language codes, server bridges
• references/transcription-config.md — Whisper streaming latency tuning, VAD config
• references/tool-calling.md — function tools, preambles, parallel calls, event taxonomy
• references/hermes-integration.md — SOUL.md injection, tool gateway dispatch, session persistence
• scripts/bridge.py — the WebRTC/WS bridge server (run from this skill)
• scripts/web/ — the browser UI (served by bridge.py)

---

Related Hermes features

This skill plays well with:

• SOUL.md / Personality — injected as session.instructions for the voice agent
• Cron scheduling — voice agent can schedule reminders via the dispatcher
• MCP integration — MCP servers available to text Hermes are also available via the voice tool dispatcher
• Session search (FTS5) — voice transcripts go into the same store and become searchable
• Voice memo transcription — separate flow for messaging platforms; this skill is for live calls

It does NOT replace Hermes's existing voice handling — it adds a new modality.