# Twilio Voice Conversation Relay > Build low-latency voice AI agents on Twilio using ConversationRelay, streaming your own LLM over a WebSocket while Twilio handles STT and TTS. - Skill: `matrixx0070/twilio-voice-conversation-relay` (Agent Skill) - Install (CLI): `npx skillmds@latest add matrixx0070/twilio-voice-conversation-relay` - Raw SKILL.md: https://api.skillmd.com/api/skills/matrixx0070/twilio-voice-conversation-relay/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: Matrixx0070 (https://skillmd.com/u/matrixx0070) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/matrixx0070/twilio-voice-conversation-relay --- ## When to use Use this when you want a real-time, barge-in-capable voice agent on a phone call and you want to bring your own LLM. Twilio's `` verb offloads speech-to-text and text-to-speech to Twilio and streams structured events to your WebSocket server; you reply with text tokens and Twilio speaks them. This is the right tool for conversational IVR replacement, AI receptionists, and outbound voice bots. **Not for:** raw audio processing where you need the media frames yourself (use `` Media Streams instead — see the aside below); multi-party bridging (see twilio-conference-calls); or recording flows (see twilio-call-recordings). ## Method 1. Point your incoming-call webhook (Voice URL on the number) at an endpoint that returns TwiML containing ``. 2. Set `url` to your `wss://` endpoint. Decide TTS/STT: pick `ttsProvider` (e.g. `ElevenLabs`, `Amazon`, `Google`) and `voice`, plus `transcriptionProvider` and `language`. Decide interruption behavior with `interruptible` and DTMF capture with `dtmfDetection`. 3. On WebSocket connect, Twilio sends a `setup` message (callSid, from, to, custom parameters). Cache session state keyed by `callSid`. 4. On each caller utterance Twilio sends a `prompt` message with the transcribed `voicePrompt`. Feed it to your LLM. 5. Stream the LLM output back as `text` token messages. Set `last: true` on the final token so Twilio knows the turn is complete and can flush TTS. 6. Decision point — barge-in: if the caller speaks while TTS is playing and `interruptible` is on, Twilio sends an `interrupt` event with `utteranceUntilInterrupt`. Stop generating, discard the truncated turn, and wait for the next `prompt`. 7. Decision point — DTMF: if `dtmfDetection` is on, key presses arrive as `dtmf` events. Route to menu logic instead of the LLM when appropriate. 8. Handoff / end: send an `end` message (or a live-agent transfer) to hand control back to TwiML. You can return new TwiML via a `` handoff to dial a human. 9. Handle `error` events: log and gracefully close. ## Example TwiML returned by your Voice webhook: ```xml ``` WebSocket handler sketch (Node): ```js import { WebSocketServer } from "ws"; const wss = new WebSocketServer({ port: 8080 }); wss.on("connection", (ws) => { let callSid; ws.on("message", async (raw) => { const msg = JSON.parse(raw); switch (msg.type) { case "setup": callSid = msg.callSid; break; case "prompt": { const stream = await llm.stream(msg.voicePrompt); for await (const token of stream) { ws.send(JSON.stringify({ type: "text", token, last: false })); } ws.send(JSON.stringify({ type: "text", token: "", last: true })); break; } case "interrupt": llm.abort(); // caller barged in; drop the rest of this turn break; case "dtmf": handleKeypad(msg.digit); break; case "error": console.error("relay error", msg.description); break; } }); }); ``` ## Pitfalls - Forgetting `last: true` on the final text message leaves Twilio waiting; the caller hears silence until the turn times out. - Not aborting your LLM stream on `interrupt` causes stale audio to play after the caller has moved on. - The WebSocket must be `wss://` (TLS). Plain `ws://` is rejected. - Custom parameters are passed via `` children and arrive only in the `setup` event — not on later messages. - `welcomeGreeting` plays before the first `prompt`; do not also send an opening line from your LLM or the caller hears it twice. - One `` per ``; `` is terminal — TwiML after it does not execute until relay ends. ## Output format Your WebSocket sends JSON messages: `{ type: "text", token, last }` for speech, plus optional control messages (`{ type: "end" }`, language switch, or a play/DTMF instruction). Twilio sends you `setup`, `prompt`, `interrupt`, `dtmf`, and `error` messages. All are UTF-8 JSON text frames. ## Reference - TwiML verb: ``. Key attributes: `url` (wss, required), `welcomeGreeting`, `ttsProvider`, `voice`, `transcriptionProvider`, `language` (BCP-47, default `en-US`), `dtmfDetection`, `interruptible`, `hints`, `profanityFilter`. Add `` children for custom setup data. - Inbound WS message types: `setup`, `prompt`, `interrupt`, `dtmf`, `error`. Outbound: `text` (with `token`, `last`), `end`, `play`, `sendDigits`, language switch. - ConversationRelay is billed on top of standard voice per-minute rates plus the chosen STT/TTS provider usage. Twilio handles the media path; you never touch raw audio. - Aside — raw media: `` (Media Streams) streams base64-encoded audio frames (mulaw, 8000 Hz, 20 ms chunks) in both directions. Use it only when you need the PCM/mulaw yourself; you then own STT/TTS. ConversationRelay is the higher-level, lower-effort path. - Media/event delivery is real-time over a single persistent WebSocket per call; Twilio closes it when the call ends or `` is redirected.