Voice AI Development

Iron Law

ALWAYS USE WEBSOCKETS FOR LIVE API — NEVER POLLING; ALWAYS USE gemini-live-2.5-flash-native-audio FOR REAL-TIME, gemini-2.5-pro-tts-preview OR gemini-2.5-flash-tts-preview FOR TTS

Voice AI Development — Gemini Live API + TTS

When to Use This Skill

Models

Live API (Real-time Voice)

• gemini-live-2.5-flash-native-audio — Primary model for real-time voice. Sub-second latency (~600ms). Barge-in, affective dialogue, multimodal (audio + camera). Always use this for real-time.

TTS Models

• gemini-2.5-pro-tts-preview — Highest fidelity. Use for audiobooks, podcasts, production voiceovers.
• gemini-2.5-flash-tts-preview — Faster and cheaper. Use for real-time TTS responses, e-learning, notifications.

TTS Model Selection Guide

Preset Voices (Live API)

Available: Aoede, Lyra, Orion (5–8 distinct steerable voices total)

• Style transfer: can mimic speaking patterns and emotions
• Cannot clone custom/external voices

SDK

# Python — ALWAYS use google-genai, NEVER google-generativeai (deprecated)
uv add google-genai

# Verify
python -c "import google.genai; print(google.genai.__version__)"

Quick Scaffold

Live API — Real-time Voice (FastAPI WebSocket)

# See reference/gemini-live-api.md for full implementation
from google import genai
from google.genai import types
import asyncio

client = genai.Client()

async def stream_voice():
    config = types.LiveConnectConfig(
        response_modalities=["AUDIO"],
        speech_config=types.SpeechConfig(
            voice_config=types.VoiceConfig(
                prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Aoede")
            )
        ),
    )
    async with client.aio.live.connect(
        model="gemini-live-2.5-flash-native-audio",
        config=config
    ) as session:
        await session.send(input="Hello, how can I help?", end_of_turn=True)
        async for response in session.receive():
            if response.data:
                yield response.data  # raw PCM bytes

TTS — Single Speaker

# See reference/gemini-tts.md for full implementation
from google import genai

client = genai.Client()
response = client.models.generate_content(
    model="gemini-2.5-flash-tts-preview",
    contents="Say this in a warm, friendly tone: Hello and welcome!",
    config={"response_modalities": ["AUDIO"]},
)
audio_bytes = response.candidates[0].content.parts[0].inline_data.data

Process

1. Choose modality — real-time streaming (Live API) vs batch TTS (TTS models)
2. Load reference files — gemini-live-api.md for streaming; gemini-tts.md for TTS
3. Pick framework — LangGraph (voice-langgraph-integration.md) or ADK (voice-adk-integration.md)
4. Implement WebSocket transport — FastAPI WebSocket endpoint proxies audio bytes bidirectionally
5. Add error handling — connection drops, rate limits, audio format validation (no silent failures)
6. Test audio pipeline — verify raw PCM bytes received and playable before adding agent logic
7. Dispatch reviewers — security-reviewer (API key handling), code-reviewer (error path coverage)

Key Patterns

Documentation Sources

Reference Files

Common Commands

# Install SDK
uv add google-genai

# Set API key (get from aistudio.google.com)
export GOOGLE_API_KEY=your_key_here

# Run FastAPI voice server
uvicorn main:app --reload --port 8000

# Test WebSocket connection (requires wscat: npm install -g wscat)
wscat -c ws://localhost:8000/ws/voice

# Test TTS endpoint
curl -X POST http://localhost:8000/tts \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello world", "model": "gemini-2.5-flash-tts-preview"}' \
  --output output.wav

# Check SDK version
python -c "import google.genai; print(google.genai.__version__)"

Error Handling

Connection drop (Live API WebSocket):
  → Catch asyncio.CancelledError and websockets.exceptions.ConnectionClosed
  → Log error with session context
  → Re-raise — do NOT silently reconnect without user awareness

Rate limit (429):
  → Log warning with retry-after header value
  → Implement exponential backoff (1s, 2s, 4s, max 32s)
  → Raise RateLimitError after max retries

Audio format error:
  → Validate input: PCM 16-bit, 16kHz, mono for Live API
  → Log format mismatch with received vs expected
  → Raise AudioFormatError — do NOT attempt to play malformed audio

Content policy block:
  → Log blocked content category (never log the content itself)
  → Return error state to caller — do NOT return silence as if successful

API key / auth (401):
  → Check GOOGLE_API_KEY env var is set
  → Log "API key missing or invalid" — never log the key value
  → Raise AuthenticationError immediately — no retry

Post-Code Review

After writing Gemini voice integration code, dispatch:

• security-reviewer — API key handling, no keys in code, audio data sanitization
• code-reviewer — WebSocket lifecycle, error path coverage, no silent failures