# Build Voice Agent > Build a production voice agent with the Patter SDK — receive or place real phone calls answered by an AI agent. Use when the user wants to make a phone bot, IVR replacement, voice receptionist, outbound caller, survey bot, customer-support voice agent, voice-AI demo, AI cold-caller, or any flow where an LLM talks over the telephone. Covers the three Patter modes — OpenAI Realtime / Realtime2 (GA, lowest latency), ElevenLabs ConvAI (turn-taking), and the STT→LLM→TTS Pipeline (mix any providers) — in both Python and TypeScript. Trigger this skill even if the user only says "phone bot", "voice AI", "call my customers", or "answer the phone with AI" without naming Patter. - Skill: `patterai/build-voice-agent` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add patterai/build-voice-agent` - Raw SKILL.md: https://api.skillmd.com/api/skills/patterai/build-voice-agent/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: PatterAI (https://skillmd.com/u/patterai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/patterai/build-voice-agent --- # Build a voice agent with Patter Patter is an open-source SDK that turns any AI agent into a phone agent. Pick one of three architectures (Realtime, ConvAI, Pipeline), describe the behaviour in a system prompt, and run a local server that connects to your Twilio or Telnyx number — no Patter Cloud, no managed service. The four-line pattern below is the entire surface. ## Architecture in one diagram ``` ┌────────────────────────────────────────────────────────────────────┐ │ Patter server (local) │ │ ┌────────────────┐ ┌───────────────────┐ ┌───────────────┐ │ │ │ Carrier WS │───▶│ Agent (engine / │───▶│ Carrier WS │ │ │ │ in (audio in) │ │ pipeline) loop │ │ out (TTS out) │ │ │ └────────────────┘ └───────────────────┘ └───────────────┘ │ │ ▲ │ │ │ │ Built-in tools: │ │ │ │ transfer_call · end_call │ │ └───────────┼──────────────────────────────────────────────┼──────────┘ │ │ │ WSS (mulaw 8kHz / pcm 16kHz) │ ▼ ▼ ┌──────────┐ ┌──────────┐ │ Twilio │ ──────── real phone call ───── │ User │ │ Telnyx │ └──────────┘ └──────────┘ ``` **You write**: system prompt, optional tools, optional guardrails, choice of engine. **Patter writes**: WebSocket framing, audio transcoding, barge-in, VAD, cost tracking, tunnel, dashboard. ## Decide the mode | Mode | When to use | Latency | Cost | |---|---|---|---| | **`OpenAIRealtime2`** (GA, default) | Default for everything. Lowest latency, single key. | ~200 ms turn | $$$ | | **`OpenAIRealtime`** | Legacy `gpt-realtime-mini`; pick only if explicitly required. | ~200 ms turn | $$ | | **`ElevenLabsConvAI`** | Turn-taking conversations (slower but better at long pauses). | ~600 ms turn | $$ | | **`Pipeline`** | Custom STT/LLM/TTS combinations. Use when the user wants Anthropic Claude / Cerebras / Whisper / a specific TTS voice. | ~800 ms turn | $-$$$$ | **Default pick**: `OpenAIRealtime2`. Switch only if the user explicitly asks for custom voice/LLM, lower cost, or a non-OpenAI provider. ## Quick start ### Python ```python import asyncio from getpatter import Patter, Twilio, OpenAIRealtime2 async def main(): phone = Patter(carrier=Twilio(), phone_number="+15550001234") agent = phone.agent( engine=OpenAIRealtime2(), system_prompt=( "You are Mia, the AI receptionist for Acme Plumbing. " "Greet the caller warmly. Help them book a service visit. " "Keep replies under two sentences." ), first_message="Hi, this is Mia at Acme Plumbing — how can I help?", ) await phone.serve(agent, tunnel=True) asyncio.run(main()) ``` ### TypeScript ```typescript import { Patter, Twilio, OpenAIRealtime2 } from "getpatter"; const phone = new Patter({ carrier: new Twilio(), phoneNumber: "+15550001234", }); const agent = phone.agent({ engine: new OpenAIRealtime2(), systemPrompt: "You are Mia, the AI receptionist for Acme Plumbing. " + "Greet the caller warmly. Help them book a service visit. " + "Keep replies under two sentences.", firstMessage: "Hi, this is Mia at Acme Plumbing — how can I help?", }); await phone.serve({ agent, tunnel: true }); ``` Both expose a tunnel URL on startup (`tunnel ready: https://abc.trycloudflare.com`). Point your Twilio number's voice webhook at that URL (see `configure-telephony`), call the number, talk to Mia. ## Pick the right mode The default works for 80% of agents. Switch when: | Want this | Pick this mode | Reference | |---|---|---| | Lowest possible latency, OpenAI voice | `OpenAIRealtime2` (default) | [references/realtime-mode.md](references/realtime-mode.md) | | Better long-pause handling, ElevenLabs voice | `ElevenLabsConvAI` | [references/convai-mode.md](references/convai-mode.md) | | Mix custom STT / LLM / TTS (Anthropic, Deepgram, Cartesia, …) | Pipeline | [references/pipeline-mode.md](references/pipeline-mode.md) | | Save cost on high-volume outbound | Pipeline with Cerebras LLM + Deepgram STT + ElevenLabs Turbo | [references/pipeline-mode.md](references/pipeline-mode.md) | | Noisy line / speakerphone (denoiser, high-pass, AGC, semantic turn detection) | Pipeline audio levers | [references/pipeline-mode.md](references/pipeline-mode.md) | Read the corresponding reference only when the user picks that mode — do not preload all three. ## Outbound calls The same `Patter` instance places outbound calls — but **`phone.call()` always needs a running server to dial through** (raises `PatterConnectionError` otherwise), and you choose how the call's lifecycle is reported: - **`wait=True` (completion-aware, since 0.6.3)** — `call()` blocks until the callee hangs up and returns a `CallResult` with the `outcome`, duration, transcript, and cost. This is what you want for scripts and campaigns: one `await`, one resolved call. - **`wait=False` (default, fire-and-forget)** — `call()` returns at the moment the carrier *dials* (not at hangup) and yields `None`/`void`. The call then lives entirely inside the running server — so something has to keep that server alive (a long-running `serve()`, or the `async with` / `await using` block below) for the conversation to continue. The completion-aware form is the recommended pattern. `async with Patter(...)` (Python) / `await using` (TypeScript) boots the local server, keeps it alive for the call's lifetime, and tears it down cleanly on exit — no dangling process, no manual `serve()` task to cancel: ```python import asyncio from getpatter import Patter, Twilio, OpenAIRealtime2 async def main(): # `async with` runs the local server for the duration of the block. async with Patter(carrier=Twilio(), phone_number="+15550001234") as phone: agent = phone.agent( engine=OpenAIRealtime2(), system_prompt="You are Mia from Acme. Keep replies under two sentences.", first_message="Hi, this is Mia from Acme.", ) result = await phone.call( to="+14155551234", agent=agent, voicemail_message="Sorry we missed you. Call back at +1...", wait=True, # block until the call ends → CallResult ) # CallResult is frozen: call_id, outcome, status, # duration_seconds, transcript, cost, metrics. print(result.outcome) # answered | voicemail | no_answer | busy | failed print(f"{result.duration_seconds:.0f}s · ${result.cost.total_usd:.4f}") asyncio.run(main()) ``` ```typescript import { Patter, Twilio, OpenAIRealtime2 } from "getpatter"; // `await using` runs the local server, then disposes it when the block exits. await using phone = new Patter({ carrier: new Twilio(), phoneNumber: "+15550001234", }); const agent = phone.agent({ engine: new OpenAIRealtime2(), systemPrompt: "You are Mia from Acme. Keep replies under two sentences.", firstMessage: "Hi, this is Mia from Acme.", }); const result = await phone.call({ to: "+14155551234", agent, voicemailMessage: "Sorry we missed you. Call back at +1...", wait: true, // block until the call ends → CallResult }); // CallResult is readonly: callId, outcome, status, // durationSeconds, transcript, cost, metrics. console.log(result.outcome); // answered | voicemail | no_answer | busy | failed console.log(`${result.durationSeconds}s · $${result.cost.totalUsd}`); ``` `wait=True` is timeout-bounded on `ring_timeout` (default 25 s), so a number that never picks up resolves to `outcome="no_answer"` rather than hanging forever. `machine_detection` is on by default since 0.6.3: Patter auto-detects voicemail, plays the call's `voicemail_message` before hanging up, and the `CallResult` comes back with `outcome="voicemail"`. A non-empty `voicemail_message` implicitly enables AMD even if you pass `machine_detection=False`. Otherwise the live person hears `first_message` and the agent starts the conversation. See `configure-telephony` for the full set of outbound options. ## System prompt — what matters for voice Voice prompts are different from chat prompts. Patter doesn't add hidden instructions, so be explicit: 1. **Identity**: "You are , calling on behalf of ." 2. **Goal**: "Help the caller book a service visit." (one line) 3. **Length rule**: "Keep every reply under two sentences." (LLMs over-explain on voice.) 4. **Stop conditions**: "If the caller wants a human, call `transfer_call`." 5. **Persona**: "Warm, professional, never robotic. Say 'mm-hm' when listening." The `first_message` is what the agent says immediately on pickup — keep it to 1 sentence under 8 seconds of audio. ## Gotchas - **Patter has no Patter Cloud as of 0.7.0** — `Patter(api_key=...)` raises `NotImplementedError`. Always use `carrier=...` + `phone_number=...`. - **Built-in tools are always present**: every agent gets `transfer_call` and `end_call` for free. You don't have to register them. - **Mulaw vs PCM audio**: Twilio carrier = mulaw 8 kHz, Telnyx = PCM 16 kHz. Patter transcodes both transparently — you never touch raw audio unless you write a pipeline hook. - **`tunnel=True` is dev-only**. For production, use a static webhook URL (your own subdomain or paid ngrok). Cloudflare quick tunnels work but occasionally have a ~3 s WSS upgrade race on first call. - **`first_message` is played before the LLM responds**. If you omit it, there's an awkward 1–2 s pause while the LLM warms up. - **Barge-in works by default** since 0.6.3 (VAD activation 0.8, deactivation 0.65 — tuned for room noise). If your callers are interrupting at the wrong moments, see [`inspect-calls-and-metrics`](../inspect-calls-and-metrics/) to read the VAD events from the call log. - **Callers cut off mid-pause?** In Pipeline mode add a semantic `turn_detector` (smart-turn or NAMO) and, on noisy lines, the `denoiser` / `high_pass_hz` / `agc` levers — see [references/pipeline-mode.md](references/pipeline-mode.md). In Realtime mode use `openai_realtime_noise_reduction="far_field"` and `realtime_turn_detection` — see [references/realtime-mode.md](references/realtime-mode.md). - **`prewarm_first_message=False` is the default** since 0.6.3 — it was briefly flipped to `True` mid-release and reverted because it conflicted with barge-in. ## Common errors | Symptom | Fix | |---|---| | `RuntimeError: Patter Cloud is not implemented` | Don't pass `api_key=`. Use `carrier=Twilio()` + `phone_number="..."`. | | `PatterConnectionError` on `call(..., wait=True)` | No running server to dial through. Wrap the call in `async with Patter(...)` / `await using`, or place it while a `serve()` task is alive. | | Agent says nothing on pickup | Missing `first_message=`. Add one. | | LLM responses are 4 paragraphs long | Add "Keep replies under two sentences" to system prompt. | | Caller hears garbled audio | Wrong audio rate. Check you're using the right carrier (mulaw 8 kHz Twilio, PCM 16 kHz Telnyx). Patter handles this; user code only breaks it via custom pipeline hooks. | | Connection drops after 30 s | Either the tunnel died (use static URL) or barge-in hung. Check `MetricsStore` for the last event. | ## Related skills - [`setup-patter`](../setup-patter/) — install + env vars (run this first if user hasn't). - [`configure-telephony`](../configure-telephony/) — set up the Twilio/Telnyx webhook. - [`add-tools-and-handoffs`](../add-tools-and-handoffs/) — custom tools, `transfer_call`, guardrails. - [`inspect-calls-and-metrics`](../inspect-calls-and-metrics/) — live dashboard, cost per call, transcript. ## References - Patter overview: · - Concepts: - Agent configuration: · - Examples: