AI Voice Bots

AI Voice Bots

Use this skill to build, ship, and tune voice bots — phone IVR, real-time speech agents, and voice-first customer interactions — using pure Python frameworks.

This skill owns the voice-specific pipeline: STT, TTS, telephony platforms, latency engineering, and voice quality. For conversation design, persona, and escalation patterns, use ai-bot-builder.

Default posture for Jul 2026: Pipecat as default framework, sub-700ms total turn latency, streaming STT→LLM→TTS pipeline, deterministic VAD, and voice quality metrics from day one. S2S (speech-to-speech) via OpenAI Realtime API or Gemini Live is available as a latency-first alternative — see Framework Selection and references/s2s-and-native-voice-apis.md.

When to Use This Skill

• Building a voice bot for phone, IVR, or real-time speech
• Choosing a telephony platform (Twilio, Vapi, Bland.ai, Retell, Telnyx, Vonage)
• Choosing a voice pipeline framework (Pipecat, LiveKit Agents, Vocode)
• Engineering latency budgets for voice (TTFB, total turn latency)
• Selecting and configuring STT/TTS providers (Deepgram, ElevenLabs, Cartesia, Azure)
• Monitoring voice quality (MOS, WER, call completion rate)
• Designing IVR flows with DTMF and voice hybrid
• Building outbound dialing campaigns

When NOT to Use This Skill

Quick Reference

Default Workflow

1. Define call flow — inbound vs outbound, IVR menu tree, conversation states.
2. Choose telephony platform — by volume, region, compliance, and API quality.
3. Choose voice pipeline framework — Pipecat (default) or LiveKit Agents.
4. Set latency budgets — per pipeline stage, total turn latency < 700ms target.
5. Select STT/TTS providers — by language support, latency, quality, and cost.
6. Integrate conversation logic — use ai-bot-builder patterns for the LLM "brain."
7. Add voice-specific guardrails — recording consent, PII in speech, barge-in safety.
8. Instrument voice quality metrics — MOS, WER, call completion, latency percentiles.
9. Load test and tune — verify latency under concurrent call load.

ASCII Flow

voice bot request
  -> call flow: inbound, outbound, IVR, or real-time agent
  -> platform choice: telephony, WebRTC, or managed voice API
  -> pipeline choice
     +-- cascading -> STT -> LLM -> TTS
     +-- speech-to-speech -> native real-time voice API
  -> latency budget and provider selection
  -> conversation brain via ai-bot-builder
  -> consent, PII, barge-in, and transfer guardrails
  -> MOS, WER, completion rate, and p95/p99 latency monitoring
  -> load test, fallback path, and launch decision

Voice Pipeline Architecture

Phone/WebRTC → Transport → STT → LLM → TTS → Transport → Phone/WebRTC
                  │          │      │      │         │
                  │          │      │      │         └── Audio codec encoding
                  │          │      │      └── Text-to-speech streaming
                  │          │      └── Conversation logic (ai-bot-builder)
                  │          └── Speech-to-text streaming
                  └── WebSocket / WebRTC / SIP

Pipeline latency budget (target: < 700ms total):

Full depth → references/voice-pipeline-architecture.md

Telephony Platform Selection

Default: Twilio for maximum control and global reach. Vapi for fastest time-to-market. Retell for managed S2S without infrastructure work.

Full comparison → references/telephony-platform-selection.md

Framework Selection

Default: Pipecat — strongest Python ecosystem, composable pipeline processors, multi-transport support, and broadest S2S provider coverage.

Use LiveKit Agents when: multi-participant calls, built-in recording, or already using LiveKit infrastructure.

S2S vs Cascading Decision Tree

Is latency < 500ms (p50) a hard requirement?
├── No → Cascading (STT→LLM→TTS)
│         Reasons: text-layer compliance, guardrails, PII redaction, logging, debugging
└── Yes → Do you need text-layer inspection?
          ├── Yes (compliance, PII, guardrails) → Cascading — latency goal may need re-negotiation
          └── No → Speech-to-speech (S2S)
                   ├── OpenAI Realtime API (GA, out of beta; `gpt-4o-realtime-preview` retired) —
                   │     `gpt-realtime-2.1` / `gpt-realtime-2.1-mini` (reasoning + tool use in both tiers,
                   │     ~25% lower p95 latency than the prior `gpt-realtime-2` generation), plus
                   │     `gpt-realtime-translate` (70+ input / 13 output languages, live speech translation) and
                   │     `gpt-realtime-whisper` (streaming STT, controllable latency/quality tradeoff).
                   │     Verify current model names before use — this line moves every few months.
                   └── Gemini Live — Google ecosystem, good for long sessions

S2S trade-offs: audio-in → audio-out bypasses the text layer entirely (~500ms vs ~700ms cascading). You lose: text-layer compliance filtering, PII detection/redaction, guardrail injection, intermediate transcript logging, and the ability to inspect model reasoning. Some teams ship S2S for the product experience and add a parallel transcript path (Deepgram streaming alongside) to recover the audit trail — but this adds cost and complexity.

Full S2S reference → references/s2s-and-native-voice-apis.md

Production Defaults

• Framework: Pipecat with streaming pipeline
• STT: Deepgram Nova-3 for transcription-only pipelines (54% WER reduction over Nova-2). For conversational pipelines, prefer Deepgram Flux (flux-general-en / flux-general-multi, multilingual GA Apr 2026) — fuses STT + turn detection in one /v2/listen call (~260ms median EoT), removing the stacked STT→VAD→endpointing layers. See references/voice-pipeline-architecture.md.
• TTS: ElevenLabs Flash v2.5 (eleven_flash_v2_5, 75ms model inference, 32 languages) for real-time conversations, or Cartesia for ultra-low latency — Sonic-3 (40ms TTFA on Turbo, independent benchmarks show higher variance) or the newer Sonic-3.5 + Ink-2 streaming STT/TTS stack (shipped Jun 2026; verify current benchmark numbers before committing). Note: eleven_turbo_v2_5 is functionally equivalent (Flash has lower average latency); Eleven v3 (GA Mar 2026) adds expressive control but is not suitable for real-time/conversational use.
• LLM: Claude Sonnet for complex conversations, Haiku for simple routing
• Transport: Twilio for phone, Daily/WebRTC for web
• Latency target: < 700ms total turn latency (p90). Top-tier: < 500ms (p50)
• Quality monitoring: MOS tracking, WER sampling, call completion rate
• Compliance: Recording consent per jurisdiction, PII redaction from transcripts

Expert Judgment: Latency Budget and Barge-In

Decompose the budget before you optimize. "700ms feels slow" is not actionable; "TTS first-chunk is 320ms of our 700ms" is. Attribute every millisecond to VAD, STT, LLM TTFB, TTS first-chunk, or network before touching code — teams that skip this step optimize the component that's easiest to change (usually the LLM prompt) instead of the one that's actually the bottleneck (usually VAD end-of-speech wait or a cold TTS connection). Full worked budget and instrumentation → references/latency-engineering.md; queueing-theory-grounded budget partitioning → references/queueing-theory-applied.md.

Barge-in is a pre-emption problem, not a VAD-tuning problem. A user talking over the bot must cancel in-flight TTS and the pending LLM stream within one residual service period (~150-300ms for a typical TTS chunk) or the interruption reads as broken, even if VAD detected it instantly. Don't use a fixed silence timeout for barge-in — set it from observed per-stage latency percentiles, and monitor barge-in success rate (bot audio actually stops within ~350ms) as its own SLO, separately from turn latency. See references/queueing-theory-applied.md (P3, A3) for the pre-emption pattern and its failure mode.

S2S vs cascading is a latency-vs-control trade, re-evaluated per release, not a one-time architecture choice. Cascading gives you the text layer (compliance, PII redaction, guardrails, debuggability) at the cost of ~150-300ms extra hops. S2S buys latency but forces either giving up the text layer or paying for it back with a parallel transcript track — which can erase the simplicity gain that made S2S attractive. Because S2S providers now ship a new generation roughly every 8-10 weeks (see references/s2s-and-native-voice-apis.md), a "cascading was necessary for latency" decision from two quarters ago may no longer hold — re-check the current-generation p50/p95 numbers before defending a standing architecture choice on stale benchmarks.

Known Traps

• proving latency with synthetic lab prompts instead of real barge-in, interruption, packet-loss, and handset-network conditions
• treating telephony acceptance as conversation success when the real failure is post-answer latency, bad turn segmentation, or TTS overlap
• mixing recording, transcript retention, PCI redaction, and consent rules across regions without one explicit policy owner
• optimizing only average latency while ignoring p95 or p99 tails that make production calls feel broken
• shipping one STT or TTS provider path with no fallback, rollback, or degraded-mode behavior for provider incidents
• S2S session-state loss on model switch: switching between S2S model versions mid-session (any provider — OpenAI Realtime, Gemini Live) drops all ephemeral session state — voice, tone configuration, conversation history, and tool state are not carried over. Resolution: persist conversation state to an external store (Redis or Postgres) after every turn; reload from the store when resuming or switching models. Do not rely on the S2S session as a state store for anything you cannot afford to lose. See references/s2s-and-native-voice-apis.md for the full session management pattern.

Common Anti-Patterns

• Batch-style voice pipelines — waiting for full utterances or full synthesis destroys turn-taking and makes the bot feel laggy
• LLM-first architecture with no deterministic call state — IVR routing, transfers, and compliance prompts need explicit state machines, not only prompt logic
• One-metric quality reporting — MOS alone or WER alone hides interruption quality, completion failures, and escalation pain
• Treating outbound voice like chat automation — dialing, consent, voicemail handling, and retry policy need channel-specific controls
• Using text-bot guardrails unchanged for speech — voice bots need barge-in, silence, DTMF, and speaking-over-user protections

Navigation

References

• references/index.md — Reference navigation map
• references/s2s-and-native-voice-apis.md — S2S vs cascading, OpenAI Realtime API, Gemini Live, session management (Jul 2026)
• references/telephony-platform-selection.md — Platform comparison
• references/voice-pipeline-architecture.md — Pipeline design
• references/pipecat-patterns.md — Pipecat deep dive
• references/livekit-agents-patterns.md — LiveKit Agents deep dive
• references/latency-engineering.md — Latency optimization
• references/voice-quality-metrics.md — Quality monitoring
• references/ivr-design.md — IVR flow design
• references/voice-safety-compliance.md — Voice compliance
• references/queueing-theory-applied.md — Queueing theory applied to voice: latency budget partitioning, jitter buffer sizing, Erlang-C IVR capacity, barge-in priority, TTS streaming targets

Assets

• assets/voice-bot-spec.md — Voice bot specification template
• assets/voice-latency-budget.md — Latency budget worksheet
• assets/voice-quality-checklist.md — Pre-launch quality gate
• assets/voice-eval-scenarios.md — End-to-end voice-agent eval scenario template

Scripts

• python3 scripts/voice_latency_audit.py --input pipeline_logs.jsonl — Pipeline latency breakdown
• python3 scripts/call_quality_scorer.py --input calls.jsonl — Call quality scoring

Data

• data/sources.json — Curated voice-specific sources

Related Skills

• ai-bot-builder — Conversation design, persona, escalation, LangGraph
• ../ai-context-layer/references/conversational-surfaces-cross-platform.md — Cross-platform composition recipe; voice section specifies the RA13 hot/cold memory tier split required for sub-300 ms turn latency
• ../ai-agents/SKILL.md — Agent architecture decisions
• ../ai-agents/references/voice-multimodal-agents.md — Voice/multimodal agent reference
• ../software-realtime/SKILL.md — WebSocket/SSE infrastructure
• ../qa-agent-testing/SKILL.md — Agent eval harnesses
• ../qa-observability/SKILL.md — Pipeline telemetry

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Start from data/sources.json for voice framework docs and provider references.
• Verify current STT/TTS provider APIs, latency benchmarks, and pricing before citing specifics.
• Telephony platform features and pricing change frequently — verify before recommending.
• If live verification is unavailable, mark provider-specific guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.