| name | build-voice-agent |
| description | Build a production voice agent with the Patter SDK — receive or place real phone calls answered by an AI agent. Use when the user wants to make a phone bot, IVR replacement, voice receptionist, outbound caller, survey bot, customer-support voice agent, voice-AI demo, AI cold-caller, or any flow where an LLM talks over the telephone. Covers the three Patter modes — OpenAI Realtime / Realtime2 (GA, lowest latency), ElevenLabs ConvAI (turn-taking), and the STT→LLM→TTS Pipeline (mix any providers) — in both Python and TypeScript. Trigger this skill even if the user only says "phone bot", "voice AI", "call my customers", or "answer the phone with AI" without naming Patter.
|
| license | MIT |
| compatibility | Requires Patter >= 0.7.0, a configured carrier (Twilio or Telnyx) and provider key (OPENAI_API_KEY for Realtime; ELEVENLABS_API_KEY for ConvAI; STT+LLM+TTS keys for Pipeline). Use the `setup-patter` skill first if the user hasn't installed and configured Patter.
|
| metadata | {"author":"patter","version":"0.1.0","parity":"both"} |
Build a voice agent with Patter
Patter is an open-source SDK that turns any AI agent into a phone agent. Pick
one of three architectures (Realtime, ConvAI, Pipeline), describe the
behaviour in a system prompt, and run a local server that connects to your
Twilio or Telnyx number — no Patter Cloud, no managed service. The four-line
pattern below is the entire surface.
Architecture in one diagram
┌────────────────────────────────────────────────────────────────────┐
│ Patter server (local) │
│ ┌────────────────┐ ┌───────────────────┐ ┌───────────────┐ │
│ │ Carrier WS │───▶│ Agent (engine / │───▶│ Carrier WS │ │
│ │ in (audio in) │ │ pipeline) loop │ │ out (TTS out) │ │
│ └────────────────┘ └───────────────────┘ └───────────────┘ │
│ ▲ │ │
│ │ Built-in tools: │ │
│ │ transfer_call · end_call │ │
└───────────┼──────────────────────────────────────────────┼──────────┘
│ │
│ WSS (mulaw 8kHz / pcm 16kHz) │
▼ ▼
┌──────────┐ ┌──────────┐
│ Twilio │ ──────── real phone call ───── │ User │
│ Telnyx │ └──────────┘
└──────────┘
You write: system prompt, optional tools, optional guardrails, choice of engine.
Patter writes: WebSocket framing, audio transcoding, barge-in, VAD, cost tracking, tunnel, dashboard.
Decide the mode
| Mode | When to use | Latency | Cost |
|---|
OpenAIRealtime2 (GA, default) | Default for everything. Lowest latency, single key. | ~200 ms turn | $$$ |
OpenAIRealtime | Legacy gpt-realtime-mini; pick only if explicitly required. | ~200 ms turn | $$ |
ElevenLabsConvAI | Turn-taking conversations (slower but better at long pauses). | ~600 ms turn | $$ |
Pipeline | Custom STT/LLM/TTS combinations. Use when the user wants Anthropic Claude / Cerebras / Whisper / a specific TTS voice. | ~800 ms turn | $-$$$$ |
Default pick: OpenAIRealtime2. Switch only if the user explicitly asks for
custom voice/LLM, lower cost, or a non-OpenAI provider.
Quick start
Python
import asyncio
from getpatter import Patter, Twilio, OpenAIRealtime2
async def main():
phone = Patter(carrier=Twilio(), phone_number="+15550001234")
agent = phone.agent(
engine=OpenAIRealtime2(),
system_prompt=(
"You are Mia, the AI receptionist for Acme Plumbing. "
"Greet the caller warmly. Help them book a service visit. "
"Keep replies under two sentences."
),
first_message="Hi, this is Mia at Acme Plumbing — how can I help?",
)
await phone.serve(agent, tunnel=True)
asyncio.run(main())
TypeScript
import { Patter, Twilio, OpenAIRealtime2 } from "getpatter";
const phone = new Patter({
carrier: new Twilio(),
phoneNumber: "+15550001234",
});
const agent = phone.agent({
engine: new OpenAIRealtime2(),
systemPrompt:
"You are Mia, the AI receptionist for Acme Plumbing. " +
"Greet the caller warmly. Help them book a service visit. " +
"Keep replies under two sentences.",
firstMessage: "Hi, this is Mia at Acme Plumbing — how can I help?",
});
await phone.serve({ agent, tunnel: true });
Both expose a tunnel URL on startup (tunnel ready: https://abc.trycloudflare.com).
Point your Twilio number's voice webhook at that URL (see configure-telephony),
call the number, talk to Mia.
Pick the right mode
The default works for 80% of agents. Switch when:
Read the corresponding reference only when the user picks that mode — do not
preload all three.
Outbound calls
The same Patter instance places outbound calls — but phone.call() always
needs a running server to dial through (raises PatterConnectionError
otherwise), and you choose how the call's lifecycle is reported:
wait=True (completion-aware, since 0.6.3) — call() blocks until the
callee hangs up and returns a CallResult with the outcome, duration,
transcript, and cost. This is what you want for scripts and campaigns: one
await, one resolved call.wait=False (default, fire-and-forget) — call() returns at the moment
the carrier dials (not at hangup) and yields None/void. The call then
lives entirely inside the running server — so something has to keep that
server alive (a long-running serve(), or the async with / await using
block below) for the conversation to continue.
The completion-aware form is the recommended pattern. async with Patter(...)
(Python) / await using (TypeScript) boots the local server, keeps it alive
for the call's lifetime, and tears it down cleanly on exit — no dangling
process, no manual serve() task to cancel:
import asyncio
from getpatter import Patter, Twilio, OpenAIRealtime2
async def main():
async with Patter(carrier=Twilio(), phone_number="+15550001234") as phone:
agent = phone.agent(
engine=OpenAIRealtime2(),
system_prompt="You are Mia from Acme. Keep replies under two sentences.",
first_message="Hi, this is Mia from Acme.",
)
result = await phone.call(
to="+14155551234",
agent=agent,
voicemail_message="Sorry we missed you. Call back at +1...",
wait=True,
)
print(result.outcome)
print(f"{result.duration_seconds:.0f}s · ${result.cost.total_usd:.4f}")
asyncio.run(main())
import { Patter, Twilio, OpenAIRealtime2 } from "getpatter";
await using phone = new Patter({
carrier: new Twilio(),
phoneNumber: "+15550001234",
});
const agent = phone.agent({
engine: new OpenAIRealtime2(),
systemPrompt: "You are Mia from Acme. Keep replies under two sentences.",
firstMessage: "Hi, this is Mia from Acme.",
});
const result = await phone.call({
to: "+14155551234",
agent,
voicemailMessage: "Sorry we missed you. Call back at +1...",
wait: true,
});
console.log(result.outcome);
console.log(`${result.durationSeconds}s · $${result.cost.totalUsd}`);
wait=True is timeout-bounded on ring_timeout (default 25 s), so a number
that never picks up resolves to outcome="no_answer" rather than hanging
forever. machine_detection is on by default since 0.6.3: Patter auto-detects
voicemail, plays the call's voicemail_message before hanging up, and the
CallResult comes back with outcome="voicemail". A non-empty
voicemail_message implicitly enables AMD even if you pass
machine_detection=False. Otherwise the live person hears first_message
and the agent starts the conversation. See configure-telephony for the
full set of outbound options.
System prompt — what matters for voice
Voice prompts are different from chat prompts. Patter doesn't add hidden
instructions, so be explicit:
- Identity: "You are , calling on behalf of ."
- Goal: "Help the caller book a service visit." (one line)
- Length rule: "Keep every reply under two sentences." (LLMs over-explain on voice.)
- Stop conditions: "If the caller wants a human, call
transfer_call."
- Persona: "Warm, professional, never robotic. Say 'mm-hm' when listening."
The first_message is what the agent says immediately on pickup — keep it
to 1 sentence under 8 seconds of audio.
Gotchas
- Patter has no Patter Cloud as of 0.7.0 —
Patter(api_key=...) raises
NotImplementedError. Always use carrier=... + phone_number=....
- Built-in tools are always present: every agent gets
transfer_call and
end_call for free. You don't have to register them.
- Mulaw vs PCM audio: Twilio carrier = mulaw 8 kHz, Telnyx = PCM 16 kHz.
Patter transcodes both transparently — you never touch raw audio unless
you write a pipeline hook.
tunnel=True is dev-only. For production, use a static webhook URL
(your own subdomain or paid ngrok). Cloudflare quick tunnels work but
occasionally have a ~3 s WSS upgrade race on first call.first_message is played before the LLM responds. If you omit it,
there's an awkward 1–2 s pause while the LLM warms up.- Barge-in works by default since 0.6.3 (VAD activation 0.8, deactivation
0.65 — tuned for room noise). If your callers are interrupting at the
wrong moments, see
inspect-calls-and-metrics
to read the VAD events from the call log.
- Callers cut off mid-pause? In Pipeline mode add a semantic
turn_detector (smart-turn or NAMO) and, on noisy lines, the
denoiser / high_pass_hz / agc levers — see
references/pipeline-mode.md. In Realtime
mode use openai_realtime_noise_reduction="far_field" and
realtime_turn_detection — see
references/realtime-mode.md.
prewarm_first_message=False is the default since 0.6.3 — it was briefly
flipped to True mid-release and reverted because it conflicted with
barge-in.
Common errors
| Symptom | Fix |
|---|
RuntimeError: Patter Cloud is not implemented | Don't pass api_key=. Use carrier=Twilio() + phone_number="...". |
PatterConnectionError on call(..., wait=True) | No running server to dial through. Wrap the call in async with Patter(...) / await using, or place it while a serve() task is alive. |
| Agent says nothing on pickup | Missing first_message=. Add one. |
| LLM responses are 4 paragraphs long | Add "Keep replies under two sentences" to system prompt. |
| Caller hears garbled audio | Wrong audio rate. Check you're using the right carrier (mulaw 8 kHz Twilio, PCM 16 kHz Telnyx). Patter handles this; user code only breaks it via custom pipeline hooks. |
| Connection drops after 30 s | Either the tunnel died (use static URL) or barge-in hung. Check MetricsStore for the last event. |
Related skills
References
