| name | elevenlabs-agents |
| description | Build conversational AI voice agents with ElevenLabs Platform. Configure agents, tools, RAG knowledge bases, agent versioning with A/B testing, and MCP security. React, React Native, or Swift SDKs. Prevents 34 documented errors.
Use when: building voice agents, AI phone systems, agent versioning/branching, MCP security, or troubleshooting @11labs deprecated, webhook errors, CSP violations, localhost allowlist, tool parsing errors.
|
| user-invocable | true |
ElevenLabs Agents Platform
Overview
ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:
- ASR (Automatic Speech Recognition) - Converts speech to text (32+ languages, sub-second latency)
- LLM (Large Language Model) - Reasoning and response generation (GPT, Claude, Gemini, custom models)
- TTS (Text-to-Speech) - Converts text to speech (5000+ voices, 31 languages, low latency)
- Turn-Taking Model - Proprietary model that handles conversation timing and interruptions
🚨 Package Updates (January 2026)
ElevenLabs migrated to new scoped packages in August 2025. Current packages:
npm install @elevenlabs/react@0.12.3
npm install @elevenlabs/client@0.12.2
npm install @elevenlabs/react-native@0.5.7
npm install @elevenlabs/elevenlabs-js@2.30.0
npm install -g @elevenlabs/agents-cli@0.6.1
DEPRECATED: @11labs/react, @11labs/client (uninstall if present)
⚠️ CRITICAL: v1 TTS models were removed on 2025-12-15. Use Turbo v2/v2.5 only.
December 2025 Updates
Widget Improvements (v0.5.5):
- Microphone permission handling improvements (better UX for permission requests)
- Text-only mode (
chat_mode: true) no longer requires microphone access
end_call system tool fix (no longer omits last message)
SDK Fixes:
- Scribe audio format parameter now correctly transmitted (v2.32.0, Jan 2026)
- React Native infinite loop fix in useEffect dependencies (v0.5.6)
- Speed parameter support in TTS overrides (v0.5.7)
- Localization support for chat UI terms (v0.12.3)
Package Selection Guide
Which ElevenLabs package should I use?
| Package | Environment | Use Case |
|---|
@elevenlabs/elevenlabs-js | Server only (Node.js) | Full API access, TTS, voices, models |
@elevenlabs/client | Browser + Server | Agents SDK, WebSocket, lightweight |
@elevenlabs/react | React apps | Conversational AI hooks |
@elevenlabs/react-native | Mobile | iOS/Android agents |
⚠️ Why elevenlabs-js doesn't work in browser:
- Depends on Node.js
child_process module (by design)
- Error:
Module not found: Can't resolve 'child_process'
- Workaround for browser API access: Create proxy server endpoint using
elevenlabs-js, call proxy from browser
Affected Frameworks:
- Next.js client components
- Vite browser builds
- Electron renderer process
- Tauri webview
Source: GitHub Issue #293
1. Quick Start
React SDK
npm install @elevenlabs/react zod
import { useConversation } from '@elevenlabs/react';
const { startConversation, stopConversation, status } = useConversation({
agentId: 'your-agent-id',
signedUrl: '/api/elevenlabs/auth',
clientTools: { },
onEvent: (event) => { },
serverLocation: 'us'
});
CLI ("Agents as Code")
npm install -g @elevenlabs/agents-cli
elevenlabs auth login
elevenlabs agents init
elevenlabs agents add "Bot" --template customer-service
elevenlabs agents push --env dev
elevenlabs agents test "Bot"
API (Programmatic)
import { ElevenLabsClient } from 'elevenlabs';
const client = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY });
const agent = await client.agents.create({
name: 'Support Bot',
conversation_config: {
agent: { prompt: { prompt: "...", llm: "gpt-4o" }, language: "en" },
tts: { model_id: "eleven_turbo_v2_5", voice_id: "your-voice-id" }
}
});
2. SDK Parameter Naming (camelCase vs snake_case)
CRITICAL: The JS SDK uses camelCase for parameters while the Python SDK and API use snake_case. Using snake_case in JS causes silent failures where parameters are ignored.
Common Parameters:
| API/Python (snake_case) | JS SDK (camelCase) |
|---|
model_id | modelId |
voice_id | voiceId |
output_format | outputFormat |
voice_settings | voiceSettings |
Example:
const stream = await elevenlabs.textToSpeech.convert(voiceId, {
model_id: "eleven_v3",
text: "Hello"
});
const stream = await elevenlabs.textToSpeech.convert(voiceId, {
modelId: "eleven_v3",
text: "Hello"
});
Tip: Always check TypeScript types for correct parameter names. This is the most common error when migrating from Python SDK.
Source: GitHub Issue #300
3. Agent Configuration
System Prompt Architecture (6 Components)
1. Personality - Identity, role, character traits
2. Environment - Communication context (phone, web, video)
3. Tone - Formality, speech patterns, verbosity
4. Goal - Objectives and success criteria
5. Guardrails - Boundaries, prohibited topics, ethical constraints
6. Tools - Available capabilities and when to use them
Template:
{
"agent": {
"prompt": {
"prompt": "Personality:\n[Agent identity and role]\n\nEnvironment:\n[Communication context]\n\nTone:\n[Speech style]\n\nGoal:\n[Primary objectives]\n\nGuardrails:\n[Boundaries and constraints]\n\nTools:\n[Available tools and usage]",
"llm": "gpt-4o",
"temperature": 0.7
}
}
}
2025 LLM Models:
gpt-5.1, gpt-5.1-2025-11-13 (Oct 2025)claude-sonnet-4-5, claude-sonnet-4-5@20250929 (Oct 2025)gemini-3-pro-preview (2025)gemini-2.5-flash-preview-09-2025 (Oct 2025)
Turn-Taking Modes
| Mode | Behavior | Best For |
|---|
| Eager | Responds quickly | Fast-paced support, quick orders |
| Normal | Balanced (default) | General customer service |
| Patient | Waits longer | Information collection, therapy |
{ "conversation_config": { "turn": { "mode": "patient" } } }
Workflows & Agent Management (2025)
Workflow Features:
- Subagent Nodes - Override prompt, voice, turn-taking per node
- Tool Nodes - Guarantee tool execution
- Edges - Conditional routing with
edge_order (determinism, Oct 2025)
{
"workflow": {
"nodes": [
{ "id": "node_1", "type": "subagent", "config": { "system_prompt": "...", "turn_eagerness": "patient" } },
{ "id": "node_2", "type": "tool", "tool_name": "transfer_to_human" }
],
"edges": [{ "from": "node_1", "to": "node_2", "condition": "escalation", "edge_order": 1 }]
}
}
Agent Management (2025):
- Agent Archiving -
archived: true field (Oct 2025)
- Agent Duplication - Clone existing agents
- Service Account API Keys - Management endpoints (Jul 2025)
Dynamic Variables
Use {{var_name}} syntax in prompts, messages, and tool parameters.
System Variables:
{{system__agent_id}}, {{system__conversation_id}}{{system__caller_id}}, {{system__called_number}} (telephony){{system__call_duration_secs}}, {{system__time_utc}}{{system__call_sid}} (Twilio only)
Custom Variables:
await client.conversations.create({
agent_id: "agent_123",
dynamic_variables: { user_name: "John", account_tier: "premium" }
});
Secret Variables: {{secret__api_key}} (headers only, never sent to LLM)
⚠️ Error: Missing variables cause "Missing required dynamic variables" - always provide all referenced variables.
3. Voice & Language Features
Multi-Voice, Pronunciation & Speed
Multi-Voice - Switch voices dynamically (adds ~200ms latency per switch):
{ "prompt": "When speaking as customer, use voice_id 'voice_abc'. As agent, use 'voice_def'." }
Pronunciation Dictionary - IPA, CMU, word substitutions (Turbo v2/v2.5 only):
{
"pronunciation_dictionary": [
{ "word": "API", "pronunciation": "ey-pee-ay", "format": "cmu" },
{ "word": "AI", "substitution": "artificial intelligence" }
]
}
PATCH Support (Aug 2025) - Update dictionaries without replacement
Speed Control - 0.7x-1.2x (use 0.9x-1.1x for natural sound):
{ "voice_settings": { "speed": 1.0 } }
Voice Cloning Best Practices:
- Clean audio (no noise, music, pops)
- Consistent microphone distance
- 1-2 minutes of audio
- Use language-matched voices (English voices fail on non-English)
Language Configuration
32+ Languages with automatic detection and in-conversation switching.
Multi-Language Presets:
{
"language_presets": [
{ "language": "en", "voice_id": "en_voice", "first_message": "Hello!" },
{ "language": "es", "voice_id": "es_voice", "first_message": "¡Hola!" }
]
}
4. Knowledge Base & RAG
Enable agents to access large knowledge bases without loading entire documents into context.
Workflow:
- Upload documents (PDF, TXT, DOCX)
- Compute RAG index (vector embeddings)
- Agent retrieves relevant chunks during conversation
Configuration:
{
"agent": { "prompt": { "knowledge_base": ["doc_id_1", "doc_id_2"] } },
"knowledge_base_config": {
"max_chunks": 5,
"vector_distance_threshold": 0.8
}
}
API Upload:
const doc = await client.knowledgeBase.upload({ file: fs.createReadStream('docs.pdf'), name: 'Docs' });
await client.knowledgeBase.computeRagIndex({ document_id: doc.id, embedding_model: 'e5_mistral_7b' });
⚠️ Gotchas: RAG adds ~500ms latency. Check index status before use - indexing can take minutes.
5. Tools (4 Types)
⚠️ BREAKING CHANGE: prompt.tools Deprecated (July 2025)
The legacy prompt.tools array was removed on July 23, 2025. All agent configurations must use the new format.
Migration Timeline:
- July 14, 2025: Legacy format still accepted
- July 15, 2025: GET responses stop including
tools field
- July 23, 2025: POST/PATCH reject
prompt.tools (active now)
Old Format (no longer works):
{
agent: {
prompt: {
tools: [{ name: "get_weather", url: "...", method: "GET" }]
}
}
}
New Format (required):
{
agent: {
prompt: {
tool_ids: ["tool_abc123"],
built_in_tools: ["end_call"]
}
}
}
Error if both used: "A request must include either prompt.tool_ids or the legacy prompt.tools array — never both"
Note: All tools from legacy format were auto-migrated to standalone tool records.
Source: Official Migration Guide
A. Client Tools (Browser/Mobile)
Execute in browser or mobile app. Tool names case-sensitive.
clientTools: {
updateCart: {
description: "Update shopping cart",
parameters: z.object({ item: z.string(), quantity: z.number() }),
handler: async ({ item, quantity }) => {
return { success: true };
}
}
}
B. Server Tools (Webhooks)
HTTP requests to external APIs. PUT support added Apr 2025.
{
"name": "get_weather",
"url": "https://api.weather.com/{{user_id}}",
"method": "GET",
"headers": { "Authorization": "Bearer {{secret__api_key}}" },
"parameters": { "type": "object", "properties": { "city": { "type": "string" } } }
}
⚠️ Secret variables only in headers (not URL/body)
2025 Features:
- transfer-to-human system tool (Apr 2025)
- tool_latency_secs tracking (Apr 2025)
⚠️ Historical Issue (Fixed Feb 2025):
Tool calling was broken with gpt-4o-mini due to an OpenAI API change. This was fixed in SDK v2.25.0+ (Feb 17, 2025). If using older SDK versions, upgrade to avoid silent tool execution failures on that model.
Source: Changelog Feb 17, 2025
C. MCP Tools (Model Context Protocol)
Connect to MCP servers for databases, IDEs, data sources.
Configuration: Dashboard → Add Custom MCP Server → Configure SSE/HTTP endpoint
Approval Modes: Always Ask | Fine-Grained | No Approval
2025 Updates:
- disable_interruptions flag (Oct 2025) - Prevents interruption during tool execution
- Tools Management Interface (Jun 2025)
⚠️ Limitations: SSE/HTTP only. Not available for Zero Retention or HIPAA.
D. System Tools
Built-in conversation control (no external APIs):
end_call, detect_language, transfer_agenttransfer_to_number (telephony)dtmf_playpad, voicemail_detection (telephony)
2025: use_out_of_band_dtmf flag for telephony integration
6. SDK Integration
useConversation Hook (React/React Native)
const { startConversation, stopConversation, status, isSpeaking } = useConversation({
agentId: 'your-agent-id',
signedUrl: '/api/auth',
clientTools: { },
onEvent: (event) => { },
onConnect/onDisconnect/onError,
serverLocation: 'us'
});
2025 Events:
agent_chat_response_part - Streaming responses (Oct 2025)agent_tool_request - Tool interaction tracking (Oct 2025)
Connection Types: WebRTC vs WebSocket
| Feature | WebSocket | WebRTC (Jul 2025 rollout) |
|---|
| Auth | signedUrl | conversationToken |
| Audio | Configurable (16k/24k/48k) | PCM_48000 (hardcoded) |
| Latency | Standard | Lower |
| Best For | Flexibility | Low-latency |
⚠️ WebRTC: Hardcoded PCM_48000, limited device switching
Platforms
- React:
@elevenlabs/react@0.12.3
- JavaScript:
@elevenlabs/client@0.12.2 - new Conversation({...})
- React Native:
@elevenlabs/react-native@0.5.7 - Expo SDK 47+, iOS/macOS (custom build required, no Expo Go)
- Swift: iOS 14.0+, macOS 11.0+, Swift 5.9+
- Embeddable Widget:
<script src="https://elevenlabs.io/convai-widget/index.js"></script>
- Widget Packages (Dec 2025):
@elevenlabs/convai-widget-embed@0.5.5 - For embedding in existing apps@elevenlabs/convai-widget-core@0.5.5 - Core widget functionality
Scribe (Real-Time Speech-to-Text - Beta 2025)
Real-time transcription with word-level timestamps. Single-use tokens, not API keys.
const { connect, startRecording, stopRecording, transcript, partialTranscript } = useScribe({
token: async () => (await fetch('/api/scribe/token')).json().then(d => d.token),
commitStrategy: 'vad',
sampleRate: 16000,
onPartialTranscript/onFinalTranscript/onError
});
Events: PARTIAL_TRANSCRIPT, FINAL_TRANSCRIPT_WITH_TIMESTAMPS, SESSION_STARTED, ERROR
⚠️ Closed Beta - requires sales contact. For agents, use Agents Platform instead (LLM + TTS + two-way interaction).
⚠️ Webhook Mode Issue:
Using speechToText.convert() with webhook: true causes SDK parsing errors. The API returns only { request_id } for webhook mode, but the SDK expects the full transcription schema.
Error Message:
ParseError: response: Missing required key "language_code"; Missing required key "text"; ...
Workaround - Use direct fetch API instead of SDK:
const formData = new FormData();
formData.append('file', audioFile);
formData.append('model_id', 'scribe_v1');
formData.append('webhook', 'true');
formData.append('webhook_id', webhookId);
const response = await fetch('https://api.elevenlabs.io/v1/speech-to-text', {
method: 'POST',
headers: { 'xi-api-key': apiKey },
body: formData,
});
const result = await response.json();
Source: GitHub Issue #232 (confirmed by maintainer)
7. Testing & Evaluation
🆕 Agent Testing Framework (Aug 2025)
Comprehensive automated testing with 9 new API endpoints for creating, managing, and executing tests.
Test Types:
- Scenario Testing - LLM-based evaluation against success criteria
- Tool Call Testing - Verify correct tool usage and parameters
- Load Testing - High-concurrency capacity testing
CLI Workflow:
elevenlabs tests add "Refund Test" --template basic-llm
{
"name": "Refund Test",
"scenario": "Customer requests refund",
"success_criteria": ["Agent acknowledges empathetically", "Verifies order details"],
"expected_tool_call": { "tool_name": "lookup_order", "parameters": { "order_id": "..." } }
}
elevenlabs tests push
elevenlabs agents test "Support Agent"
9 New API Endpoints (Aug 2025):
POST /v1/convai/tests - Create testGET /v1/convai/tests/:id - Retrieve testPATCH /v1/convai/tests/:id - Update testDELETE /v1/convai/tests/:id - Delete testPOST /v1/convai/tests/:id/execute - Execute testGET /v1/convai/test-invocations - List invocations (pagination, agent filtering)POST /v1/convai/test-invocations/:id/resubmit - Resubmit failed testGET /v1/convai/test-results/:id - Get resultsGET /v1/convai/test-results/:id/debug - Detailed debugging info
Test Invocation Listing (Oct 2025):
const invocations = await client.convai.testInvocations.list({
agent_id: 'agent_123',
page_size: 30,
cursor: 'next_page_cursor'
});
Programmatic Testing:
const simulation = await client.agents.simulate({
agent_id: 'agent_123',
scenario: 'Refund request',
user_messages: ["I want a refund", "Order #12345"],
success_criteria: ["Acknowledges request", "Verifies order"]
});
console.log('Passed:', simulation.passed);
Agent Tracking (Oct 2025): Tests now include agent_id association for better organization
8. Analytics & Monitoring
2025 Features:
- Custom Dashboard Charts (Apr 2025) - Display evaluation criteria metrics over time
- Call History Filtering (Apr 2025) -
call_start_before_unix parameter
- Multi-Voice History - Separate conversation history by voice
- LLM Cost Tracking - Per agent/conversation costs with
aggregation_interval (hour/day/week/month)
- Tool Latency (Apr 2025) -
tool_latency_secs tracking
- Usage Metrics - minutes_used, request_count, ttfb_avg, ttfb_p95
Conversation Analysis: Success evaluation (LLM-based), data collection fields, post-call webhooks
Access: Dashboard → Analytics | Post-call Webhooks | API
9. Privacy & Compliance
Data Retention: 2 years default (GDPR). Configure: { "transcripts": { "retention_days": 730 }, "audio": { "retention_days": 2190 } }
Encryption: TLS 1.3 (transit), AES-256 (rest)
Regional: serverLocation: 'eu-residency' | 'us' | 'global' | 'in-residency'
Zero Retention Mode: Immediate deletion (no history, analytics, webhooks, or MCP)
Compliance: GDPR (1-2 years), HIPAA (6 years), SOC 2 (automatic encryption)
10. Cost Optimization
LLM Caching: Up to 90% savings on repeated inputs. { "caching": { "enabled": true, "ttl_seconds": 3600 } }
Model Swapping: GPT-5.1, GPT-4o/mini, Claude Sonnet 4.5, Gemini 3 Pro/2.5 Flash (2025 models)
Burst Pricing: 3x concurrency limit at 2x cost. { "burst_pricing_enabled": true }
11. Advanced Features
2025 Platform Updates:
- Azure OpenAI (Jul 2025) - Custom LLM with Azure-hosted models (requires API version field)
- Genesys Output Variables (Jul 2025) - Enhanced call analytics
- LLMReasoningEffort "none" (Oct 2025) - Control model reasoning behavior
- Streaming Voice Previews (Jul 2025) - Real-time voice generation
- pcm_48000 audio format (Apr 2025) - New output format support
Events: audio, transcript, agent_response, tool_call, agent_chat_response_part (streaming, Oct 2025), agent_tool_request (Oct 2025), conversation_state
Custom Models: Bring your own LLM (OpenAI-compatible endpoints). { "llm_config": { "custom": { "endpoint": "...", "api_key": "{{secret__key}}" } } }
Post-Call Webhooks: HMAC verification required. Return 200 or auto-disable after 10 failures. Payload includes conversation_id, transcript, analysis.
Chat Mode: Text-only (no ASR/TTS). { "chat_mode": true }. Saves ~200ms + costs.
Telephony: SIP (sip-static.rtc.elevenlabs.io), Twilio native, Vonage, RingCentral. 2025: Twilio keypad fix (Jul), SIP TLS remote_domains validation (Oct)
12. CLI & DevOps ("Agents as Code")
Installation & Auth:
npm install -g @elevenlabs/agents-cli@0.6.1
elevenlabs auth login
elevenlabs auth residency eu-residency
export ELEVENLABS_API_KEY=your-api-key
Project Structure: agents.json, tools.json, tests.json + agent_configs/, tool_configs/, test_configs/
Key Commands:
elevenlabs agents init
elevenlabs agents add "Bot" --template customer-service
elevenlabs agents push --env prod --dry-run
elevenlabs agents push --env prod
elevenlabs agents pull
elevenlabs agents test "Bot"
elevenlabs tools add-webhook "Weather" --config-path tool_configs/weather.json
elevenlabs tools push
elevenlabs tests add "Test" --template basic-llm
elevenlabs tests push
Multi-Environment: Create agent.dev.json, agent.staging.json, agent.prod.json for overrides
CI/CD: GitHub Actions with --dry-run validation before deploy
.gitignore: .env, .elevenlabs/, *.secret.json
13. Common Errors & Solutions (27 Documented)
Error 1: Missing Required Dynamic Variables
Cause: Variables referenced in prompts not provided at conversation start
Solution: Provide all variables in dynamic_variables: { user_name: "John", ... }
Error 2: Case-Sensitive Tool Names
Cause: Tool name mismatch (case-sensitive)
Solution: Ensure tool_ids: ["orderLookup"] matches name: "orderLookup" exactly
Error 3: Webhook Authentication Failures
Cause: Incorrect HMAC signature, not returning 200, or 10+ failures
Solution: Verify hmac = crypto.createHmac('sha256', SECRET).update(payload).digest('hex') and return 200
⚠️ Header Name: Use ElevenLabs-Signature (NOT X-ElevenLabs-Signature - no X- prefix!)
Error 4: Voice Consistency Issues
Cause: Background noise, inconsistent mic distance, extreme volumes in training
Solution: Use clean audio, consistent distance, avoid extremes
Error 5: Wrong Language Voice
Cause: English-trained voice for non-English language
Solution: Use language-matched voices: { "language": "es", "voice_id": "spanish_voice" }
Error 6: Restricted API Keys Not Supported (CLI)
Cause: CLI doesn't support restricted API keys
Solution: Use unrestricted API key for CLI
Error 7: Agent Configuration Push Conflicts
Cause: Hash-based change detection missed modification
Solution: elevenlabs agents init --override + elevenlabs agents pull + push
Error 8: Tool Parameter Schema Mismatch
Cause: Schema doesn't match usage
Solution: Add clear descriptions: "description": "Order ID (format: ORD-12345)"
Error 9: RAG Index Not Ready
Cause: Index still computing (takes minutes)
Solution: Check index.status === 'ready' before using
Error 10: WebSocket Protocol Error (1002)
Cause: Network instability, incompatible browser, or firewall issues
Symptoms:
Error receiving message: received 1002 (protocol error)
Error sending user audio chunk: received 1002 (protocol error)
WebSocket is already in CLOSING or CLOSED state
Connection cycles: Disconnected → Connected → Disconnected rapidly
Solution:
- Use WebRTC instead of WebSocket for better stability:
connectionType: 'webrtc'
- Implement reconnection logic with exponential backoff
- Check network stability and firewall rules (port restrictions)
- Test on different networks/browsers to isolate the issue
Source: GitHub Issue #134
Error 11: 401 Unauthorized in Production
Cause: Agent visibility or API key config
Solution: Check visibility (public/private), verify API key in prod, check allowlist
Error 12: Allowlist Connection Errors
Cause: Allowlist enabled but using shared link, OR localhost validation bug
Symptoms:
Host is not supported
Host is not valid or supported
Host is not in insights whitelist
WebSocket is already in CLOSING or CLOSED state
Solution:
- Configure allowlist domains in dashboard or disable for testing
- Localhost workaround: Use
127.0.0.1:3000 instead of localhost:3000
⚠️ Localhost Validation Bug:
The dashboard has inconsistent validation for localhost URLs:
- ❌
localhost:3000 → Rejected (should be valid)
- ❌
http://localhost:3000 → Rejected (protocol not allowed)
- ❌
localhost:3000/voice-chat → Rejected (paths not allowed)
- ✅
www.localhost:3000 → Accepted (invalid but accepted!)
- ✅
127.0.0.1:3000 → Accepted (use this for local dev)
Source: GitHub Issue #320
Error 13: Workflow Infinite Loops
Cause: Edge conditions creating loops
Solution: Add max iteration limits, test all paths, explicit exit conditions
Error 14: Burst Pricing Not Enabled
Cause: Burst not enabled in settings
Solution: { "call_limits": { "burst_pricing_enabled": true } }
Error 15: MCP Server Timeout
Cause: MCP server slow/unreachable
Solution: Check URL accessible, verify transport (SSE/HTTP), check auth, monitor logs
Error 16: First Message Cutoff on Android
Cause: Android needs time to switch audio mode
Solution: connectionDelay: { android: 3_000, ios: 0 } (3s for audio routing)
Error 17: CSP (Content Security Policy) Violations
Cause: Strict CSP blocks blob: URLs. SDK uses Audio Worklets loaded as blobs
Solution: Self-host worklets:
cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/- Configure:
workletPaths: { 'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js', 'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js' }
- Update CSP:
script-src 'self' https://elevenlabs.io; worker-src 'self';
Gotcha: Update worklets when upgrading @elevenlabs/client
Error 18: Webhook Payload - Null Message on Tool Calls
Cause: Schema expects message: string but ElevenLabs sends null when agent makes tool calls
Solution: Use z.string().nullable() for message field in Zod schemas
message: z.string()
message: z.string().nullable()
Real payload example:
{ "role": "agent", "message": null, "tool_calls": [{ "tool_name": "my_tool", ... }] }
Error 19: Webhook Payload - call_successful is String, Not Boolean
Cause: Schema expects call_successful: boolean but ElevenLabs sends "success" or "failure" strings
Solution: Accept both types and convert for database storage
call_successful: z.union([z.boolean(), z.string()]).optional()
function parseCallSuccessful(value: unknown): boolean | undefined {
if (value === undefined || value === null) return undefined
if (typeof value === 'boolean') return value
if (typeof value === 'string') return value.toLowerCase() === 'success'
return undefined
}
Error 20: Webhook Schema Validation Fails Silently
Cause: Real ElevenLabs payloads have many undocumented fields that strict schemas reject
Undocumented fields in transcript turns:
agent_metadata, multivoice_message, llm_override, rag_retrieval_infollm_usage, interrupted, original_message, source_medium
Solution: Add all as .optional() with z.any() for fields you don't process
Debugging tip: Use https://webhook.site to capture real payloads, then test schema locally
Error 21: Webhook Cost Field is Credits, NOT USD
Cause: metadata.cost contains ElevenLabs credits, not USD dollars. Displaying this directly shows wildly wrong values (e.g., "$78.0000" when actual cost is ~$0.003)
Solution: Extract actual USD from metadata.charging.llm_price instead
cost: metadata?.cost
const charging = metadata?.charging as any
cost: charging?.llm_price ?? null
Real payload structure:
{
"metadata": {
"cost": 78,
"charging": {
"llm_price": 0.0036188999999999995,
"llm_charge": 18,
"call_charge": 60,
"tier": "pro"
}
}
}
Note: llm_price only covers LLM costs. Audio costs may require separate calculation based on your plan.
Error 22: User Context Available But Not Extracted
Cause: Webhook contains authenticated user info from widget but code doesn't extract it
Solution: Extract dynamic_variables from conversation_initiation_client_data
const dynamicVars = data.conversation_initiation_client_data?.dynamic_variables
const callerName = dynamicVars?.user_name || null
const callerEmail = dynamicVars?.user_email || null
const currentPage = dynamicVars?.current_page || null
Payload example:
{
"conversation_initiation_client_data": {
"dynamic_variables": {
"user_name": "Jeremy Dawes",
"user_email": "jeremy@jezweb.net",
"current_page": "/dashboard/calls"
}
}
}
Error 23: Data Collection Results Available But Not Displayed
Cause: ElevenLabs agents can collect structured data during calls (configured in agent settings). This data is stored in analysis.data_collection_results but often not parsed/displayed in UI.
Solution: Parse the JSON and display collected fields with their values and rationales
const dataCollectionResults = analysis?.dataCollectionResults
? JSON.parse(analysis.dataCollectionResults)
: null
Object.entries(dataCollectionResults).forEach(([key, data]) => {
console.log(`${key}: ${data.value} (${data.rationale})`)
})
Payload example:
{
"data_collection_results": {
"customer_name": { "value": "John Smith", "rationale": "Customer stated their name" },
"intent": { "value": "billing_inquiry", "rationale": "Asking about invoice" },
"callback_number": { "value": "+61400123456", "rationale": "Provided for callback" }
}
}
Error 24: Evaluation Criteria Results Available But Not Displayed
Cause: Custom success criteria (configured in agent) produce results in analysis.evaluation_criteria_results but often not parsed/displayed
Solution: Parse and show pass/fail status with rationales
const evaluationResults = analysis?.evaluationCriteriaResults
? JSON.parse(analysis.evaluationCriteriaResults)
: null
Object.entries(evaluationResults).forEach(([key, data]) => {
const passed = data.result === 'success' || data.result === true
console.log(`${key}: ${passed ? 'PASS' : 'FAIL'} - ${data.rationale}`)
})
Payload example:
{
"evaluation_criteria_results": {
"verified_identity": { "result": "success", "rationale": "Customer verified DOB" },
"resolved_issue": { "result": "failure", "rationale": "Escalated to human" }
}
}
Error 25: Feedback Rating Available But Not Extracted
Cause: User can provide thumbs up/down feedback. Stored in metadata.feedback.thumb_rating but not extracted
Solution: Extract and store the rating (1 = thumbs up, -1 = thumbs down)
const feedback = metadata?.feedback as any
const feedbackRating = feedback?.thumb_rating ?? null
const likes = feedback?.likes
const dislikes = feedback?.dislikes
Payload example:
{
"metadata": {
"feedback": {
"thumb_rating": 1,
"likes": ["helpful", "natural"],
"dislikes": []
}
}
}
Error 26: Per-Turn Metadata Not Extracted (interrupted, source_medium, rag_retrieval_info)
Cause: Each transcript turn has valuable metadata that's often ignored
Solution: Store these fields per message for analytics and debugging
const turnAny = turn as any
const messageData = {
interrupted: turnAny.interrupted ?? null,
sourceMedium: turnAny.source_medium ?? null,
originalMessage: turnAny.original_message ?? null,
ragRetrievalInfo: turnAny.rag_retrieval_info
? JSON.stringify(turnAny.rag_retrieval_info)
: null,
}
Use cases:
interrupted: true → User spoke over agent (UX insight)source_medium → Analytics by channelrag_retrieval_info → Debug/improve knowledge base retrieval
Error 27: Upcoming Audio Flags (August 2025)
Cause: Three new boolean fields coming in August 2025 webhooks that may break schemas
Solution: Add these fields to schemas now (as optional) to be ready
has_audio: boolean
has_user_audio: boolean
has_response_audio: boolean
const schema = z.object({
has_audio: z.boolean().optional(),
has_user_audio: z.boolean().optional(),
has_response_audio: z.boolean().optional(),
})
Note: These match the existing fields in the GET Conversation API response
Error 28: Tool Parsing Fails When Tool Not Found
Cause: Calling conversations.get(id) when conversation contains tool_results where the tool was deleted/not found
Error Message:
Error: response -> transcript -> [11] -> tool_results -> [0] -> type:
Expected string. Received null.;
response -> transcript -> [11] -> tool_results -> [0] -> type:
[Variant 1] Expected "system". Received null.;
response -> transcript -> [11] -> tool_results -> [0] -> type:
[Variant 2] Expected "workflow". Received null.
Solution:
- SDK fix needed - SDK should handle null tool_results.type gracefully
- Workaround for users:
- Ensure all referenced tools exist before deleting them
- Wrap
conversation.get() in try-catch until SDK is fixed
try {
const conversation = await client.conversationalAi.conversations.get(id);
} catch (error) {
console.error('Tool parsing error - conversation may reference deleted tools');
}
Source: GitHub Issue #268
Error 29: SDK Parameter Naming Confusion (snake_case vs camelCase)
Cause: Using snake_case parameters (from API/Python SDK docs) in JS SDK, which expects camelCase
Symptoms: Parameters silently ignored, wrong model/voice used, no error messages
Common Mistakes:
convert(voiceId, { model_id: "eleven_v3" })
convert(voiceId, { modelId: "eleven_v3" })
Solution: Always use camelCase for JS SDK parameters. Check TypeScript types if unsure.
Affected Parameters: model_id, voice_id, output_format, voice_settings, and all API parameters
Source: GitHub Issue #300
Error 30: Webhook Mode ParseError with speechToText.convert()
Cause: SDK expects full transcription response but webhook mode returns only { request_id }
Error Message:
ParseError: Missing required key "language_code"; Missing required key "text"; ...
Solution: Use direct fetch API instead of SDK for webhook mode:
const formData = new FormData();
formData.append('file', audioFile);
formData.append('model_id', 'scribe_v1');
formData.append('webhook', 'true');
formData.append('webhook_id', webhookId);
const response = await fetch('https://api.elevenlabs.io/v1/speech-to-text', {
method: 'POST',
headers: { 'xi-api-key': apiKey },
body: formData,
});
const result = await response.json();
Source: GitHub Issue #232
Error 31: Package Not Compatible with Browser/Web
Cause: Using @elevenlabs/elevenlabs-js in browser/client environments (depends on Node.js child_process)
Error Message:
Module not found: Can't resolve 'child_process'
Affected Frameworks:
- Next.js client components
- Vite browser builds
- Electron renderer process
- Tauri webview
Solution:
- For browser/web: Use
@elevenlabs/client or @elevenlabs/react instead
- For full API access in browser: Create proxy server endpoint using
elevenlabs-js, call from browser
- For Electron/Tauri: Use
elevenlabs-js in main process only, not renderer
Note: This is by design - elevenlabs-js is server-only
Source: GitHub Issue #293
Error 32: prompt.tools Deprecated - POST/PATCH Rejected
Cause: Using legacy prompt.tools array field after July 23, 2025 cutoff
Error Message:
A request must include either prompt.tool_ids or the legacy prompt.tools array — never both
Solution: Migrate to new format:
{ agent: { prompt: { tools: [...] } } }
{
agent: {
prompt: {
tool_ids: ["tool_abc123"],
built_in_tools: ["end_call"]
}
}
}
Note: All legacy tools were auto-migrated to standalone records. Just update your configuration references.
Source: Official Migration Guide
Error 33: GPT-4o Mini Tool Calling Broken (Fixed Feb 2025)
Cause: OpenAI API breaking change affected gpt-4o-mini tool execution (historical issue)
Symptoms: Tools silently fail to execute, no error messages
Solution: Upgrade to SDK v2.25.0+ (released Feb 17, 2025). If using older SDK versions, upgrade or avoid gpt-4o-mini for tool-based workflows.
Source: Changelog Feb 17, 2025
Error 34: Scribe Audio Format Parameter Not Transmitted (Fixed v2.32.0)
Cause: WebSocket URI wasn't including audio_format parameter even when specified (historical issue)
Solution: Upgrade to @elevenlabs/elevenlabs-js@2.32.0 or later (released Jan 19, 2026)
Source: GitHub PR #319
14. Agent Versioning (Jan 2026)
ElevenLabs introduced Agent Versioning in January 2026, enabling git-like version control for conversational AI agents. This allows safe experimentation, A/B testing, and gradual rollouts.
Core Concepts
| Concept | ID Format | Description |
|---|
| Version | agtvrsn_xxxx | Immutable snapshot of agent config at a point in time |
| Branch | agtbrch_xxxx | Isolated development path (like git branches) |
| Draft | Per-user/branch | Work-in-progress changes before committing |
| Deployment | Traffic splits | A/B testing with percentage-based routing |
Enabling Versioning
const agent = await client.conversationalAi.agents.update({
agentId: 'your-agent-id',
enableVersioningIfNotEnabled: true
});
⚠️ Note: Once enabled, versioning cannot be disabled on an agent.
Branch Management
const branch = await client.conversationalAi.agents.branches.create({
agentId: 'your-agent-id',
parentVersionId: 'agtvrsn_xxxx',
name: 'experiment-v2'
});
const branches = await client.conversationalAi.agents.branches.list({
agentId: 'your-agent-id'
});
await client.conversationalAi.agents.branches.delete({
agentId: 'your-agent-id',
branchId: 'agtbrch_xxxx'
});
Traffic Deployment (A/B Testing)
Route traffic between branches using percentage splits:
const deployment = await client.conversationalAi.agents.deployments.create({
agentId: 'your-agent-id',
deployments: [
{ branchId: 'agtbrch_main', percentage: 90 },
{ branchId: 'agtbrch_xxxx', percentage: 10 }
]
});
const status = await client.conversationalAi.agents.deployments.get({
agentId: 'your-agent-id'
});
Use Cases:
- A/B Testing - Test new prompts on 10% of traffic before full rollout
- Gradual Rollouts - Increase traffic incrementally (10% → 25% → 50% → 100%)
- Quick Rollback - Route 100% back to stable branch if issues detected
Merging Branches
const merge = await client.conversationalAi.agents.branches.merge({
agentId: 'your-agent-id',
sourceBranchId: 'agtbrch_xxxx',
targetBranchId: 'agtbrch_main',
archiveSourceBranch: true
});
Working with Drafts
Drafts are per-user, per-branch work-in-progress states:
const draft = await client.conversationalAi.agents.drafts.get({
agentId: 'your-agent-id',
branchId: 'agtbrch_xxxx'
});
await client.conversationalAi.agents.drafts.update({
agentId: 'your-agent-id',
branchId: 'agtbrch_xxxx',
conversationConfig: {
agent: { prompt: { prompt: 'Updated system prompt...' } }
}
});
const version = await client.conversationalAi.agents.drafts.commit({
agentId: 'your-agent-id',
branchId: 'agtbrch_xxxx',
message: 'Improved greeting flow'
});
Best Practices
- Always test on branch first - Never experiment directly on production traffic
- Use descriptive branch names -
feature-multilang, fix-timeout-handling
- Start with small traffic splits - Begin at 5-10%, monitor metrics, then increase
- Archive merged branches - Keep repository clean
- Commit messages - Use clear messages for version history
Source: Agent Versioning Docs
15. MCP Security & Guardrails
When connecting MCP (Model Context Protocol) servers to ElevenLabs agents, security is critical. MCP tools can access databases, APIs, and sensitive data.
Tool Approval Modes
| Mode | Behavior | Use When |
|---|
| Always Ask | Explicit approval for every tool execution | Default - recommended for most cases |
| Fine-Grained | Auto-approve trusted ops, require approval for sensitive | Established, trusted MCP servers |
| No Approval | All tool executions auto-approved | Only thoroughly vetted, internal servers |
Configuration:
{
"mcp_config": {
"server_url": "https://your-mcp-server.com",
"approval_mode": "always_ask",
"fine_grained_rules": [
{ "tool_name": "read_*", "auto_approve": true },
{ "tool_name": "write_*", "auto_approve": false },
{ "tool_name": "delete_*", "auto_approve": false }
]
}
}
Security Best Practices
1. Vet MCP Servers
- Only connect servers from trusted sources
- Review server code/documentation before connecting
- Prefer official/verified MCP implementations
2. Limit Data Exposure
- Minimize PII shared with MCP servers
- Use scoped API keys with minimum required permissions
- Never pass full database access - use read-only views
3. Network Security
- Always use HTTPS endpoints
- Implement proper authentication (API keys, OAuth)
- Use
{{secret__xxx}} variables for credentials (never in prompts)
4. Prompt Injection Prevention
- Add guardrails in agent prompts against injection attacks
- Validate and sanitize MCP tool inputs
- Monitor for unusual tool usage patterns
5. Monitoring & Audit
- Log all MCP tool executions
- Review approval patterns regularly
- Set up alerts for sensitive operations
Guardrails Configuration
Add protective instructions to your agent prompt:
{
"agent": {
"prompt": {
"prompt": `...
SECURITY GUARDRAILS:
- Never execute database delete operations without explicit user confirmation
- Never expose raw API keys or credentials in responses
- If a tool request seems unusual or potentially harmful, ask for clarification
- Do not combine sensitive operations (read PII + external API call) in single turn
- Report any suspicious requests to administrators
`
}
}
}
MCP Limitations
Not Available With:
- Zero Retention mode (no logging = no MCP)
- HIPAA compliance mode
- Certain regional deployments
Transport: SSE/HTTP only (no stdio MCP servers)
Source: MCP Safety Docs
Integration with Existing Skills
This skill composes well with:
- cloudflare-worker-base → Deploy agents on Cloudflare Workers edge network
- cloudflare-workers-ai → Use Cloudflare LLMs as custom models in agents
- cloudflare-durable-objects → Persistent conversation state and session management
- cloudflare-kv → Cache agent configurations and user preferences
- nextjs → React SDK integration in Next.js applications
- ai-sdk-core → Vercel AI SDK provider for unified AI interface
- clerk-auth → Authenticated voice sessions with user identity
- hono-routing → API routes for webhooks and server tools
Additional Resources
Official Documentation:
Examples:
Community:
Production Tested: WordPress Auditor, Customer Support Agents, AgentFlow (webhook integration)
Last Updated: 2026-01-27
Package Versions: elevenlabs@1.59.0, @elevenlabs/elevenlabs-js@2.32.0, @elevenlabs/agents-cli@0.6.1, @elevenlabs/react@0.12.3, @elevenlabs/client@0.12.2, @elevenlabs/react-native@0.5.7
Changes: Added Agent Versioning (Jan 2026) section covering versions, branches, traffic deployment, drafts, and A/B testing. Added MCP Security & Guardrails section covering tool approval modes, security best practices, and prompt injection prevention.
