voice-agent

# Deepgram Voice Agent API Audio in, audio out, one connection. Deepgram runs the listen, think, and speak stages and the turn-taking between them. Your code streams the user's audio, plays the agent's audio, and answers function calls. [1][2] ## Decide first: the Voice Agent API or an orchestrator Both paths are supported; Deepgram publishes guides for LiveKit Agents and Pipecat. Choose by who should own the pipeline. | Build on the Voice Agent API when | Use an orchestrator (LiveKit Agents, Pipecat, Vapi, Retell) with Deepgram STT and TTS underneath when | |---|---| | You want one WebSocket and no pipeline code. End-of-turn detection, barge-in, and the handoffs between stages are handled in-process. [2] | You already run that framework, or you need its transport (for example WebRTC rooms) and client libraries. | | A Deepgram-managed LLM (OpenAI, Anthropic, Google, NVIDIA) billed through your Deepgram account is fine, or you point `think.endpoint` at your own OpenAI-compatible endpoint. [6] | You need per-stage control the agent does not expose: your own LLM loop, a TTS vendor Deepgram does not proxy, custom voice activity detection, or your own turn logic. | | Your tools can run in your client or behind an HTTP endpoint you own (`FunctionCallRequest` / `FunctionCallResponse`). [9][10] | Your tools live inside the framework's agent runtime. | For the orchestrator path, load the `examples` skill (LiveKit, Pipecat) and the SDK `conversational-stt` and `text-to-speech` skills. Deepgram's Pipecat guide runs Flux STT and Flux TTS (`flux-alexis-en`) underneath. The LiveKit guide runs three stages: LiveKit Inference (`deepgram/aura-2` voice `thalia`, hosted and billed through LiveKit Cloud with no Deepgram API key), the Deepgram plugin with your own key on `nova-3` and `aura-2-thalia-en`, then `STTv2` `flux-general-en` and `TTSv2` `flux-alexis-en` as the Flux STT and Flux TTS swap. [14] The rest of this skill covers the Voice Agent API path. ## First request The agent host is `agent.deepgram.com`; `api.deepgram.com` serves the other APIs. This call lists the LLM models Deepgram can run for you; check a `think.provider.model` value here before it goes into `Settings`. The endpoint is public: the `Authorization` header is optional and the call answers 200 without it, so it also confirms the host is reachable before you spend a key on the socket. [6] ```bash curl -s https://agent.deepgram.com/v1/agent/settings/think/models \ -H "Authorization: Token $DEEPGRAM_API_KEY" # header optional on this endpoint # 200 {"models":[{"id":"gpt-4o-mini","name":"...","provider":"open_ai"}, ...]} ``` Then open the WebSocket to `wss://agent.deepgram.com/v1/agent/converse` with the same `Authorization: Token <key>` header. The URL takes no query parameters; all configuration goes in the `Settings` message. [1][3][41] Regional endpoints are `wss://api.eu.deepgram.com`, `wss://api.au.deepgram.com`, and `wss://api.in.deepgram.com`, each on the same `/v1/agent/converse` path. [1][37] A browser cannot set headers on a WebSocket, so mint a short-lived JWT server-side with `POST https://api.deepgram.com/v1/auth/grant` (needs a Member-scope key; the TTL defaults to 30 seconds and `ttl_seconds` sets it longer; tokens from this endpoint do not work with the Manage APIs) and pass the token as the `Sec-WebSocket-Protocol` value on the handshake, the way the Browser Agent SDK does. The token only has to be valid at the handshake. [4] ## Connect, configure, stream 1. Connect. Wait for `{"type":"Welcome","request_id":"..."}`. Send nothing before it. [5] 2. Send one `Settings` message. Wait for `{"type":"SettingsApplied"}`. Send no audio and no inject messages before it. [5] 3. Stream raw audio as binary WebSocket frames. Play the binary frames you receive. JSON events arrive as text frames on the same socket, so branch on frame type first. [5] ```json { "type": "Settings", "audio": { "input": { "encoding": "linear16", "sample_rate": 16000 }, "output": { "encoding": "linear16", "sample_rate": 24000, "container": "none" } }, "agent": { "listen": { "provider": { "type": "deepgram", "version": "v2", "model": "flux-general-en" } }, "think": { "provider": { "type": "open_ai", "model": "gpt-4o-mini", "temperature": 0.7 }, "prompt": "You are a concise phone assistant. Reply in one or two sentences.", "functions": [ { "name": "get_weather", "description": "Current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string" } }, "required": ["location"] } } ] }, "speak": { "provider": { "type": "deepgram", "version": "v2", "model": "flux-alexis-en" } }, "greeting": "Hi, how can I help?" } } ``` Field notes, from the configure and model pages [3][6][7][8]: - `listen`: Flux STT (`flux-general-en`, or `flux-general-multi` with `language_hints`) requires `"version": "v2"` and gives model-integrated end-of-turn detection. Nova (`nova-3`) uses `v1`, the default, and adds `smart_format` and `language`. Drop `version` with a Flux STT model and the agent falls back to the v1 endpoint, where `flux-general-en` is not a valid model. The top-level `agent.language` is deprecated; set `listen.provider.language` (and, for third-party TTS, `speak.provider.language`) instead. [7][20][3] - `think`: `provider.type` is `open_ai`, `anthropic`, `google`, or `nvidia` (managed; `endpoint` optional) or `groq` or `aws_bedrock` (bring your own; `endpoint` required). [6] - `google`: `provider.version` picks the API: `ai-studio-v1beta` (AI Studio; `v1beta` is an alias) or `gemini-enterprise-agent-v1` (Gemini Enterprise Agent, regional endpoints, fewer models). Omitted, it defaults to `ai-studio-v1beta` on `agent.deepgram.com`, `api.au.deepgram.com`, and `api.in.deepgram.com`, and to `gemini-enterprise-agent-v1` on `api.eu.deepgram.com`. [37] - `aws_bedrock` authenticates with `provider.credentials` (`type` `iam`, or `sts` plus `session_token`, with `region`, `access_key_id`, and `secret_access_key`) and points `endpoint.url` at `https://bedrock-runtime.{region}.amazonaws.com/`. [6] - Bring your own LLM by keeping `type: open_ai` and setting `endpoint.url` to any OpenAI Chat Completions-compatible URL, with `endpoint.headers` for its auth. [6] - Pass an array of providers to get an ordered fallback chain: a request that fails or times out produces a `THINK_REQUEST_FAILED` warning and goes to the next provider, every turn starts again from the first, and when all of them fail the session gets a `FAILED_TO_THINK` error and the turn produces no reply. Managed-LLM prompts are limited to 25,000 characters; a longer prompt is truncated with a `PROMPT_TOO_LONG` warning, not rejected. [6][13] - `nvidia`: send the model string that `GET /v1/agent/settings/think/models` (the `curl` under First request) returns, currently `nvidia/nemotron-3.5-lightning-30b-a3b`. - `speak`: `"version": "v2"` selects Flux TTS (`flux-{voice}-{language}`); `v1`, the default when you name a provider, selects Aura (`aura-2-thalia-en`). Omit `agent.speak` entirely and you get Flux TTS with `flux-kit-en`. [8][3] - Flux TTS streams raw audio only: `encoding` must be `linear16`, `mulaw`, or `alaw` and `container` must be `none`. The full `audio.output.encoding` enum is `linear16`, `mulaw`, `alaw`, `mp3`, `opus`, `flac`, or `aac`, `container` is `none`, `wav`, or `ogg`, and `audio.output.bitrate` is bits per second; the compressed encodings, a container such as `wav`, and `bitrate` are Aura only, and Flux TTS answers any of them with `INVALID_SETTINGS`. [38][3] - `provider.speed` (default `1.0`) is `0.5` to `1.5` in `0.05` steps on Flux TTS and any value from `0.7` to `1.5` on Aura; a value the family does not accept ends the session with `FAILED_TO_SPEAK`. `provider.expressivity` (whole numbers `-2` to `2`, default `0`) is Flux TTS (`v2`) only and fixed for the session; it is beta and `0` is the only value validated for production. [34] - Third-party TTS (`open_ai`, `eleven_labs`, `cartesia`, `aws_polly`) takes an `endpoint` with `url` and `headers`, and `wss` URLs are accepted for Eleven Labs only; `aws_polly` also requires `credentials` (`type` `sts` or `iam`, with `region`, `access_key_id`, `secret_access_key`, and `session_token` for STS). Deepgram-managed Cartesia (`type: cartesia` with no `endpoint`) is the exception. [8][12] - `provider.language` is for Cartesia and Eleven Labs; Deepgram voices carry their language in the model name (`flux-kit-en`, `aura-2-sirio-es`) and a Deepgram provider given `language` is rejected with `UNPARSABLE_CLIENT_MESSAGE`. [3][8] - `speak` also takes an array of providers as an ordered fallback chain: a failed request produces a `SPEAK_REQUEST_FAILED` warning and goes to the next provider, and when all of them fail the session ends with `FAILED_TO_SPEAK`. [8][12] - `agent.context.messages` replays earlier turns as `{"type":"History","role":"user","content":"..."}` or `{"type":"History","function_calls":[{"id":"...","name":"...","client_side":false,"arguments":"...","response":"..."}]}` so a new session continues an old one. [39] While `Settings.flags.history` is `true` (the default) the server sends `History` messages in the same two shapes; set it to `false` to turn them off. [3][35] - Other knobs: `think.provider.reasoning_mode` accepts `low`, `medium`, or `high` on OpenAI reasoning models such as `gpt-5` and `gpt-5-mini` [3]. `think.context_length` (`max` or a character count; custom `think.endpoint` only) and the top-level `tags`, `experimental`, and `mip_opt_out` have their ranges in `references/agent.md` in the `api` skill. [12] - Conversational Mode (`agent.think_conversational.provider`: backchanneling, presence checks, frustration detection) is invite-only Early Access; a `Settings` message that names it from an unenrolled project is rejected with `UNPARSABLE_CLIENT_MESSAGE`. [36] ## Reusable agent configurations `Settings.agent` is either the full `agent` object above or a Reusable Agent Configuration UUID string, the same `agent: "YOUR_AGENT_ID"` form the Browser Agent SDK takes. Create one with `POST https://api.deepgram.com/v1/projects/{project_id}/agents` and a body whose `config` is the JSON string of the `agent` block (plus optional `metadata`); the response's `agent_id` is the UUID. `GET .../agents` lists them, `GET .../agents/{agent_id}` reads one, `PUT .../agents/{agent_id}` changes `metadata` only (`config` is immutable; delete and recreate to change it), and `DELETE .../agents/{agent_id}` removes it. Deleting a configuration that a running service still references breaks that service, so move its sessions to a new UUID first. A `Settings` message with a UUID that does not resolve ends the session with `INVALID_AGENT_ID`; `AGENT_ID_NOT_SUPPORTED` means the server does not resolve UUIDs at all (a self-hosted build in unauthenticated mode). [32][13] Template variables, at `POST/GET /v1/projects/{project_id}/agent-variables` and `GET/PATCH/DELETE .../agent-variables/{variable_id}`, hold values a `config` references by key in the `DG_<NAME>` form (uppercase letters, digits, `_`, `-`), written unquoted inside the JSON string. A variable can stand in for any JSON value, a whole provider object included, and `is_sensitive` must be `false`. Every project member can read configurations and variables, so keep API keys and passwords out of them. Full request and response schemas: `references/agent.md` in the `api` skill. [32] ## Message lifecycle There is no separate logging API. For per-session observability, persist every non-audio frame in both directions, keyed to the `request_id` from `Welcome`. [40][41] | From | Message | What to do | |---|---|---| | server | `Welcome` | Send `Settings`. [5] | | server | `SettingsApplied` | Start streaming audio. [5] | | server | `UserStartedSpeaking` | Barge-in. Stop playback now and discard every buffered agent audio frame. [11] | | server | `ConversationText` (`role` is `user` or `assistant`, `content`; with `flux-general-multi`, user-role messages also carry `languages_hinted` and `languages`, BCP-47 arrays, the second sorted by word count descending) | Show the transcript. [11] | | server | `AgentThinking` (`content`) | Optional status. The LLM is working, possibly choosing a function. [11] | | server | `FunctionCallRequest` | See the next section. [9] | | server | `FunctionCallCancelled` (`functions[]` with `id`, `name`) | The user started speaking again. Stop work on each `id` and do not send a `FunctionCallResponse` for it; a late one is dropped. [33] | | server | `AgentStartedSpeaking` | The reply's audio is starting. [12] | | server | `LatencyReport` | Per-turn latency breakdown, sent automatically after each turn: `stt_latency`, `ttt_token_latency`, `ttt_text_latency`, `ttt_tool_latency`, `ttt_thinking_latency`, `tts_latency`, `total_latency`. All are floats in seconds and each is optional, so read them defensively. [31] | | server | binary frames | Agent audio. Queue it for playback. [5] | | server | `AgentAudioDone` | Last chunk sent. The user may still be hearing buffered audio, so treat your output queue as the end-of-playback signal rather than this event. [11] | | server | `Error` / `Warning` (`code`, `description`) | `Error` ends the session; reconnect. `MAXIMUM_SESSION_LENGTH_REACHED` is the 2-hour close; `FAILED_TO_THINK` and `FAILED_TO_SPEAK` mean every provider in the chain failed. `Warning` is informational: `MAXIMUM_SESSION_LENGTH_APPROACHING` at 1:55, `PROMPT_TOO_LONG`, `THINK_REQUEST_FAILED`, and `SPEAK_REQUEST_FAILED`; the `think` and `speak` field notes give each one's meaning. [13] | | client | `KeepAlive` | Only while you are not sending audio, one every 8 seconds. It does not extend the 2-hour session limit. [15] | Mid-call updates, each acknowledged by a matching `*Updated` event [16]: