realtime

# ElevenLabs Realtime Realtime models are ElevenLabs models built for low-latency voice conversation. There are two ways to use them: - **An ElevenAgents agent** (an agent ID, the dashboard, or an `elevenlabs agents` project with `agent_configs/`): set the agent's Voice mode to Realtime with `conversation_config.realtime_model`, and ElevenLabs manages its LLM, TTS model, and ASR settings. See [Run an agent on a Realtime model](#run-an-agent-on-a-realtime-model). - **A browser app built on OpenAI's Realtime SDK**: keep the SDK and point it at the OpenAI-compatible ElevenLabs Realtime API, `wss://api.elevenlabs.io/v1/convai/realtime`. See [Migrate an app from OpenAI Realtime](#migrate-an-app-from-openai-realtime). When the user is starting from scratch and it's unclear which they want, ask. An agent brings a managed system prompt, tools, knowledge base, widget, and telephony; the Realtime API is the quicker path for code that already speaks OpenAI's realtime events. For agent features beyond the voice mode (prompt, tools, workflows, widget), use the `agents` skill. ## Models | Model | Model ID | Optimized for | | ----------------------- | ------------------------- | ----------------------------------------------------------------------------------------- | | Eleven Realtime v1 Mini | `eleven_realtime_v1_mini` | Lowest latency and cost. The default. | | Eleven Realtime v1 Max | `eleven_realtime_v1_max` | Reasons before it responds, for complex instructions and multi-step tool use. Costs more. | Start with Mini. Choose Max when the work involves long or complex instructions or multi-step tool calls and Mini's responses aren't good enough, or when an app being migrated was on a larger OpenAI model for reasoning-heavy work. Agents add compliance and data residency constraints on the choice; see [Choose the model](#step-2-choose-the-model). ## Run an agent on a Realtime model Every voice agent has a Voice mode. In **Custom** mode the agent's LLM, TTS model, and ASR settings are chosen individually. In **Realtime** mode the agent runs on a Realtime model and ElevenLabs manages those settings. The mode is one field: `conversation_config.realtime_model` set to a model ID means Realtime, `null` means Custom. The system prompt, first message, voice, languages, temperature, tools, knowledge base, and conversation flow settings (turn eagerness, timeouts, interruptions) work the same in both modes. ### Before you start - These steps call the ElevenLabs API. The CLI and SDKs read `ELEVENLABS_API_KEY` from the environment; if it isn't set, use the `setup-api-key` skill. Never ask the user to paste the key into the conversation. - Realtime models are rolling out gradually. If the API rejects `realtime_model` with `Realtime models are not available for this workspace.`, the workspace doesn't have access yet. Stop and tell the user; no configuration change fixes it. ### Step 1: Check the agent can use Realtime For an existing agent, fetch its configuration first (`elevenlabs agents pull --agent <agent-id>`, or `agents.get` in the SDK). Keep the agent in Custom mode, and tell the user why, if any of these apply: | Blocker | Where to look | | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Custom LLM or Amazon Bedrock | `conversation_config.agent.prompt.llm` is `custom-llm` or a Bedrock model | | Created through another ElevenLabs product, such as Speech Engine | Ask the user if unsure; only ElevenAgents agents can use a Realtime model | | Needs a specific LLM or TTS model | Ask the user whether the current `agent.prompt.llm` or `tts.model_id` is a requirement | | Workflow nodes override a [managed setting](#managed-settings) or `agent.prompt.max_tokens` | Overrides on the nodes under the agent's `workflow` | | Transfers to or from agents in Custom mode | `transfer_to_agent` tools and `agent_transfer` workflow nodes on this agent, and on any agent that transfers into this one | | Text-only agent | `conversation_config.conversation.text_only` is `true`; Realtime models are built for voice | A workflow override can be removed and a transfer partner can be switched to Realtime too, but only with the user's agreement: both change how other parts of their setup behave. ### Step 2: Choose the model Start with `eleven_realtime_v1_mini`. Choose `eleven_realtime_v1_max` when the agent follows long or complex instructions or relies on multi-step tool calls and Mini isn't good enough, or when the agent must comply with Zero Retention Mode or HIPAA. Then check availability: | Environment | `eleven_realtime_v1_mini` | `eleven_realtime_v1_max` | | ----------------------------- | ------------------------- | -------------------------- | | US | Available | Available | | EU data residency | Available | Available | | India data residency | Available | Not available | | Singapore data residency | Available | Not available | | Zero Retention Mode and HIPAA | Not available | Available in the US and EU | Zero Retention Mode applies whether it's enforced on the workspace or enabled on the agent. Ask the user when the workspace's data residency or compliance settings aren't clear from the project. If neither model is available for the agent's environment, keep it in Custom mode. Max also needs `agent.prompt.max_tokens` of at least 1024, or `-1` for no limit. ### Step 3: Set the model On an existing agent, change only `realtime_model`; leave the rest of the configuration alone. With the CLI, pull the agent, set the field in its file under `agent_configs/`, and push: ```bash elevenlabs agents pull --agent <agent-id> ``` ```json { "conversation_config": { "realtime_model": "eleven_realtime_v1_mini" } } ``` ```bash elevenlabs agents push --agent <agent-id> ``` With the SDKs: ```python from elevenlabs import ElevenLabs elevenlabs = ElevenLabs() elevenlabs.conversational_ai.agents.update( agent_id="<agent-id>", conversation_config={ "realtime_model": "eleven_realtime_v1_mini", }, ) ``` ```typescript import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js"; const elevenlabs = new ElevenLabsClient(); await elevenlabs.conversationalAi.agents.update("<agent-id>", { conversationConfig: { realtimeModel: "eleven_realtime_v1_mini", }, }); ``` For a new agent, put `realtime_model` (`realtimeModel` in TypeScript) in the `conversation_config` passed to `agents.create`, alongside the prompt, first message, and voice. Don't pick an LLM or TTS model for it; a Realtime agent ignores them. To turn Realtime off, set `realtime_model` to `null` (`None` in Python). The agent goes back to the LLM, TTS model, and ASR settings stored on it. ### Managed settings While a Realtime model is set, the agent ignores the stored values of these fields in `conversation_config`. They come back into use when the agent switches to Custom, so don't clear them, and don't edit them to tune a Realtime agent: | Setting | Field | | ------------------- | ----------------------------------------- | | LLM | `agent.prompt.llm` | | Reasoning effort | `agent.prompt.reasoning_effort` | | Thinking budget | `agent.prompt.thinking_budget` | | Parallel tool calls | `agent.prompt.enable_parallel_tool_calls` | | Cascade timeout | `agent.prompt.cascade_timeout_seconds` | | TTS model | `tts.model_id` | | Expressive mode | `tts.expressive_mode` | | ASR provider | `asr.provider` | `agent.prompt.max_tokens` keeps applying. Temperature does too, but the dashboard hides it in Realtime mode, so change it through the CLI or API. If the Realtime model errors or doesn't respond in time, the agent falls back to its backup LLMs (LLM cascading); the Realtime model sets how long it waits. ### Step 4: Publish and verify The Step 1 rules are checked when the agent is published, not when a draft is saved: a draft that breaks one saves, then publishing fails with an error naming the setting to change. 1. Fetch the agent and confirm `conversation_config.realtime_model` holds the chosen model. If the push, update, or the user's publish in the dashboard fails, match the error below. 2. Find every agent that transfers into this one and confirm it's in Realtime mode too. A Custom-mode agent's transfer into a Realtime agent is only caught when that agent is next published; until then the transfer is rejected mid-conversation and the caller stays with the current agent. 3. Have the user hold a test conversation. ### Errors | Error | Cause and fix | | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `Realtime models are not available for this workspace.` | The workspace doesn't have access yet. Nothing to fix in the configuration. | | `Transfers between default and Realtime model modes are not supported.` | A transfer to or from a Custom-mode agent ("default" means Custom). Put both agents in the same mode. | | An error naming the Realtime model | The model isn't available under the agent's compliance or data residency settings. See Step 2. | | Publishing fails naming a setting | One of the Step 1 blockers or Max's token limit. Change that setting, or keep the agent in Custom. | ### Pricing A Realtime agent is billed the standard ElevenAgents per-minute rate plus its input, cached input, and output tokens at the Realtime model's rates; Max costs more per token than Mini. Point the user to https://elevenlabs.io/pricing rather than quoting rates. ## Migrate an app from OpenAI Realtime ElevenLabs Realtime speaks the same wire protocol as OpenAI's GA Realtime API: the same client and server events over WebSocket. An app built on OpenAI's Realtime SDK keeps that SDK; the migration changes where credentials are minted, which URL the transport connects to, and the model, voice, and session configuration. The one structural change: ElevenLabs Realtime is WebSocket-only. In a browser, `RealtimeSession` defaults to WebRTC, which wires the microphone and speaker for you. After switching to the WebSocket transport, the app has to capture and play audio itself. This covers apps where the browser opens the realtime connection. It doesn't cover apps where a server opens the connection and streams audio to its clients over its own protocol, such as a bespoke WebSocket relay or a telephony bridge. Connecting to ElevenLabs Realtime from a server should work, but this skill has no steps for it. Tell the user so, and suggest they open an issue or a pull request at https://github.com/elevenlabs/skills describing their setup, rather than guessing at the server-side migration. ### Step 0: Inventory the app Find every place the app touches OpenAI Realtime before editing anything: ```bash grep -rnE "api\.openai\.com/v1/realtime|client_secrets|realtime/sessions|@openai/agents-realtime|@openai/agents/realtime|gpt-realtime|gpt-4o(-mini)?-realtime|OpenAIRealtimeWebRTC|OpenAIRealtimeWebSocket|RealtimeSession|OPENAI_API_KEY" --include='*.{ts,tsx,js,jsx,mjs,py}' . ``` Classify what you find, because each part migrates differently: - **Credential minting**: a server route that calls `/v1/realtime/client_secrets` (GA) or `/v1/realtime/sessions` (beta). Migrate in Step 1. - **Browser session**: `RealtimeSession`, `OpenAIRealtimeWebRTC`, `OpenAIRealtimeWebSocket`, or a hand-rolled `WebSocket`/`RTCPeerConnection`. Migrate in Steps 2 to 4. - **Server-side realtime connections**: a Node or Python process that opens the realtime socket itself, for example a relay that streams audio to clients over its own WebSocket protocol, a telephony bridge, `openai` Python's `client.realtime.connect()`, or a `RealtimeSession` running in Node with a full API key. Not covered; see the scope above. If the app also has a browser session, migrate only that part, and only with the user's agreement. - **Features beyond a voice conversation**: function tools, handoffs, guardrails, MCP servers, image input, SIP. ElevenLabs documents the voice conversation flow only. Leave this code in place, list it for the user as unverified against ElevenLabs, and have them test it. `OPENAI_API_KEY` often serves other OpenAI calls too. Remove it only if nothing else uses it. ### Step 1: Mint the client secret from ElevenLabs The browser never sees the ElevenLabs API key. A server route mints a short-lived client secret and returns only its `value`. | What | OpenAI | ElevenLabs | | --------------- | -------------------------------------------------------- | ------------------------------------------------------------------ | | URL | `POST https://api.openai.com/v1/realtime/client_secrets` | `POST https://api.elevenlabs.io/v1/convai/realtime/client_secrets` | | Auth header | `Authorization: Bearer $OPENAI_API_KEY` | `xi-api-key: $ELEVENLABS_API_KEY` |