2068 Openai API Routing 3626ac97

OpenAI API Routing and Proxy Patterns

voice-mode uses the OpenAI SDK, which means you can redirect all API traffic through a custom router using the standard OPENAI_BASE_URL environment variable - no voice-mode specific code or configuration needed.

Why OpenAI API Compatibility Matters

By using only standard OpenAI endpoints, voice-mode becomes a simple API client that can work with any OpenAI-compatible service. This design enables:

1. Service Agnostic: Works with OpenAI, Azure OpenAI, local services, or any compatible API
2. Simple Configuration: Just change the BASE_URL environment variable
3. External Control: All routing logic lives outside voice-mode
4. Future Proof: New providers can be added without code changes

Routing Patterns

1. Simple Proxy with Fallback

# nginx configuration example
upstream openai_primary {
    server api.openai.com:443;
}

upstream openai_fallback {
    server 127.0.0.1:8880;  # Local Kokoro TTS
}

location /v1/audio/speech {
    proxy_pass https://openai_primary;
    proxy_intercept_errors on;
    error_page 502 503 504 = @fallback;
}

location @fallback {
    proxy_pass http://openai_fallback;
}

2. Cost-Optimized Router

# Python routing proxy example
from fastapi import FastAPI, Request
import httpx

app = FastAPI()

@app.post("/v1/audio/speech")
async def route_tts(request: Request):
    body = await request.json()
    
    # Route based on model
    if body.get("model") == "tts-1":
        # Use local Kokoro for standard TTS
        async with httpx.AsyncClient() as client:
            return await client.post(
                "http://127.0.0.1:8880/v1/audio/speech",
                json=body
            )
    else:
        # Use OpenAI for advanced models
        async with httpx.AsyncClient() as client:
            return await client.post(
                "https://api.openai.com/v1/audio/speech",
                json=body,
                headers={"Authorization": request.headers.get("Authorization")}
            )

3. Voice-Specific Routing

// Node.js Express router
app.post('/v1/audio/speech', async (req, res) => {
    const { voice, model } = req.body;
    
    // Route Sky voice to Kokoro, others to OpenAI
    if (voice === 'af_sky') {
        const response = await fetch('http://127.0.0.1:8880/v1/audio/speech', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ ...req.body, model: 'kokoro-v0.87' })
        });
        res.send(await response.arrayBuffer());
    } else {
        // Forward to OpenAI
        const response = await fetch('https://api.openai.com/v1/audio/speech', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': req.headers.authorization
            },
            body: JSON.stringify(req.body)
        });
        res.send(await response.arrayBuffer());
    }
});

4. Load Balancer with Health Checks

# HAProxy configuration
global
    maxconn 4096

defaults
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

backend tts_servers
    balance roundrobin
    option httpchk GET /health
    
    server openai api.openai.com:443 ssl verify none check
    server kokoro1 127.0.0.1:8880 check
    server kokoro2 127.0.0.1:8881 check backup

Configuration

Simple Global Routing

The easiest approach is to use the OpenAI SDK's built-in support:

# Route ALL OpenAI API calls through your proxy
export OPENAI_BASE_URL="https://router.example.com/v1"
export OPENAI_API_KEY="your-key"

That's it! The OpenAI SDK automatically uses this base URL for all API calls.

Note: If STT_BASE_URL or TTS_BASE_URL are not explicitly set, voice-mode defaults to using the OpenAI API. When OPENAI_BASE_URL is set, the OpenAI SDK will use it automatically for these default cases.

Provider-Specific Routing

For more granular control, voice-mode also supports provider-specific endpoints:

# Use different endpoints for STT and TTS
export STT_BASE_URL="http://127.0.0.1:2022/v1"  # Local Whisper
export TTS_BASE_URL="http://127.0.0.1:8880/v1"  # Local Kokoro

Your router receives all requests and can then:

• Route to different providers
• Add authentication
• Log requests
• Implement rate limiting
• Cache responses
• Transform requests/responses

Benefits

1. Zero Code Changes: voice-mode doesn't need to know about routing logic
2. Centralized Control: Manage all API routing in one place
3. Dynamic Switching: Change routing rules without restarting voice-mode
4. Multi-Service: One router can handle STT, TTS, and other OpenAI APIs
5. Monitoring: Add metrics, logging, and observability at the proxy layer

Example: LiteLLM Proxy

LiteLLM provides a ready-made proxy for OpenAI-compatible routing:

# Install LiteLLM
pip install litellm[proxy]

# Configure proxy
cat > litellm_config.yaml << EOF
model_list:
  - model_name: tts-1
    litellm_params:
      model: openai/tts-1
      api_key: $OPENAI_API_KEY
  - model_name: whisper-1
    litellm_params:
      model: openai/whisper-1
      api_key: $OPENAI_API_KEY
EOF

# Start proxy
litellm --config litellm_config.yaml --port 9000

# Configure voice-mode
export TTS_BASE_URL="http://127.0.0.1:9000"
export STT_BASE_URL="http://127.0.0.1:9000"

This architecture ensures voice-mode remains simple while enabling sophisticated deployment patterns through external routing layers.