Model Router

Model Router

When to Use

Trigger phrases:

• "model router"

• "When managing multiple LLM providers (OpenAI, Anthropic, Google, etc"

• "When optimizing for cost vs quality tradeoffs"

• "When implementing fallback chains for reliability"

• When managing multiple LLM providers (OpenAI, Anthropic, Google, etc.)

• When optimizing for cost vs quality tradeoffs

• When implementing fallback chains for reliability

• When routing by task type (code to Claude, vision to Gemini, etc.)

When NOT to Use

• For single-provider setups (just use that provider SDK)
• For local-only inference (use Ollama skills)

Overview

Model routing is the intelligence layer between your application and the diverse landscape of large language model providers. Rather than hardcoding a single provider, a model router evaluates each request against dynamic criteria — task type, required capability, latency budget, cost ceiling, and current provider health — to select the optimal model at runtime. This transforms LLM integration from brittle point-to-point connections into a resilient, cost-aware system.

The core problem routing solves is that no single model dominates every dimension. Claude excels at code generation and structured reasoning, GPT-4o handles creative writing and multimodal tasks, Gemini-2.5-flash delivers the lowest latency for high-throughput workloads, and open-source models offer cost advantages for batch processing. Without routing, teams either overspend on premium models for trivial tasks or degrade quality by using cheap models everywhere. A router captures these trade-offs as programmable rules.

Production-grade routers implement three layers: static routing (model-per-task mapping), dynamic routing (cost/latency-aware selection with real-time metrics), and fallback chains (primary → secondary → tertiary on failure). Each layer adds resilience. When OpenAI experiences an outage, traffic shifts to Anthropic. When latency spikes on Claude, the router demotes it in favor of Gemini without any developer intervention.

Beyond simple selection, routers track cumulative cost, enforce per-workspace budgets, collect latency percentiles, and surface provider reliability trends. These metrics enable continuous optimization — A/B testing between model versions, gradual rollouts of new providers, and automated downgrades when spending exceeds thresholds.

A well-designed router decouples the application from individual model SDKs. Teams change model assignments through configuration, not code deploys. New providers plug in via a common interface. This architectural boundary is what allows fast iteration as the model landscape evolves.

Workflow

1. Inventory available models — Catalog every model accessible across your providers. Record each model's capabilities (code, vision, reasoning, multilingual), pricing tier (input/output per 1K tokens), context window size, rate limits, and known limitations. Maintain this inventory as a structured configuration file, not scattered environment variables.

2. Define routing rules — Create a routing table that maps task types and constraints to ordered model lists. Rules can be simple (code→Claude, vision→Gemini) or compound (reasoning tasks under 4K tokens → GPT-4o-mini for cost savings, over 4K → Claude Opus for quality). Express constraints as cost ceilings, latency SLOs, and required capability flags.

3. Implement provider abstraction — Wrap each provider's SDK behind a common interface with methods for chat completion, streaming, embedding, and token counting. Normalize error types (rate limit, authentication, server error, timeout) so the router handles them uniformly. Implement retry with exponential backoff at this layer.

4. Build fallback chains — For each route, define a primary, secondary, and tertiary model. The router tries the primary first. On non-recoverable errors (auth failure, 4xx), it skips to the next. On transient errors (5xx, timeout, rate limit), it retries with backoff before falling through. Log every fallback event for reliability monitoring.

5. Instrument cost and latency tracking — Record per-request metrics: model used, input/output tokens, latency, cost, and fallback depth. Aggregate into periodic summaries (daily cost by provider, p50/p95/p99 latency per model, success rate per provider). Store in a time-series database or structured log sink for trend analysis.

6. Configure health checks and circuit breakers — Implement a health probe that periodically tests each provider with a lightweight request. Track consecutive failures. When failures exceed a threshold, trip a circuit breaker that excludes that provider from routing until a cooldown period expires. Circuit breaker state changes should trigger alerts.

7. Optimize continuously — Run A/B experiments comparing model versions within the same route (e.g., 50% traffic to GPT-4o, 50% to GPT-4.1). Measure quality via downstream signals (user ratings, task completion rate). Adjust routing weights based on empirical data. Add new models to low-risk routes first, then promote as confidence grows.

Anti-Rationalization Table

Code Example (TypeScript)

const routes: Record<string, string[]> = {
  code: ['claude-sonnet-4-20250514', 'gpt-4o', 'gemini-2.5-pro'],
  vision: ['gemini-2.5-pro', 'gpt-4o', 'claude-sonnet-4-20250514'],
  fast: ['gemini-2.5-flash', 'gpt-4o-mini', 'claude-haiku'],
  creative: ['gpt-4o', 'claude-opus-4-20250514', 'gemini-2.5-pro'],
};

async function routeRequest(task: string, prompt: string) {
  const models = routes[task] || routes.fast;
  for (const model of models) {
    try {
      return await callModel(model, prompt);
    } catch (e) {
      console.warn(`${model} failed, trying next...`);
    }
  }
  throw new Error('All models failed');
}

Code Example (Python)

import time
import logging
from typing import Any

# Router configuration with cost tracking
ROUTES = {
    "code": ["claude-sonnet-4", "gpt-4o", "gemini-2.5-pro"],
    "vision": ["gemini-2.5-pro", "gpt-4o", "claude-sonnet-4"],
    "fast": ["gemini-2.5-flash", "gpt-4o-mini"],
    "creative": ["gpt-4o", "claude-opus-4", "gemini-2.5-pro"],
}

COST_PER_1K_TOKENS = {
    "gpt-4o": (0.01, 0.03),        # (input, output)
    "gpt-4o-mini": (0.0015, 0.006),
    "claude-sonnet-4": (0.003, 0.015),
    "claude-opus-4": (0.015, 0.075),
    "gemini-2.5-pro": (0.01, 0.04),
    "gemini-2.5-flash": (0.0005, 0.0015),
}

class CostAwareRouter:
    """Routes requests with cost tracking and latency budgets."""

    def __init__(self, latency_budget_s: float = 5.0):
        self.latency_budget_s = latency_budget_s
        self.stats: dict[str, list[dict[str, Any]]] = {}

    async def route(self, task: str, prompt: str) -> str:
        models = ROUTES.get(task, ROUTES["fast"])
        for model in models:
            start = time.monotonic()
            try:
                result = await self._call_model(model, prompt)
                elapsed = time.monotonic() - start
                self._record(model, elapsed, "success")
                if elapsed > self.latency_budget_s:
                    logging.warning("%s within budget but exceeded latency SLO", model)
                return result
            except Exception as exc:
                elapsed = time.monotonic() - start
                self._record(model, elapsed, f"fail:{type(exc).__name__}")
                logging.warning("Fallback from %s after %.1fs", model, elapsed)
        raise RuntimeError("All models exhausted")

    def _record(self, model: str, latency: float, status: str) -> None:
        self.stats.setdefault(model, []).append({
            "latency": round(latency, 3),
            "status": status,
            "timestamp": time.time(),
        })

    def report(self) -> dict[str, Any]:
        summary = {}
        for model, records in self.stats.items():
            success = [r for r in records if r["status"] == "success"]
            summary[model] = {
                "total": len(records),
                "success": len(success),
                "avg_latency": round(sum(r["latency"] for r in success) / len(success), 3) if success else None,
            }
        return summary

Setup / Configuration

Provider Credentials

Store provider API keys in environment variables or a secrets manager, never in code. Define a common interface that all providers implement:

import os
from abc import ABC, abstractmethod

class LLMProvider(ABC):
    @abstractmethod
    async def chat(self, model: str, messages: list, **kwargs) -> dict:
        ...

Model Inventory Config

Maintain a structured configuration file that maps model IDs to their capabilities, pricing, and rate limits:

# models.yaml
models:
  claude-sonnet-4:
    provider: anthropic
    capabilities: [code, reasoning, structured-output]
    context_window: 200000
    input_price_per_1k: 0.003
    output_price_per_1k: 0.015
    rate_limit_rpm: 1000
  gpt-4o:
    provider: openai
    capabilities: [vision, creative-writing, multilingual]
    context_window: 128000
    input_price_per_1k: 0.01
    output_price_per_1k: 0.03
    rate_limit_rpm: 500
  gemini-2.5-flash:
    provider: google
    capabilities: [fast, multilingual, vision]
    context_window: 1048576
    input_price_per_1k: 0.0005
    output_price_per_1k: 0.0015
    rate_limit_rpm: 2000

Routing Table

Define routing rules in a separate config section that maps task types to ordered model lists with constraint annotations:

routes:
  code:
    models: [claude-sonnet-4, gpt-4o, gemini-2.5-pro]
    max_cost_per_request: 0.05
    required_capabilities: [code, reasoning]
  vision:
    models: [gemini-2.5-pro, gpt-4o, claude-sonnet-4]
    required_capabilities: [vision, multilingual]
  fast:
    models: [gemini-2.5-flash, gpt-4o-mini]
    max_latency_ms: 2000
    max_cost_per_request: 0.003

Dependencies

Install the provider SDKs your router supports:

pip install openai anthropic google-generativeai
# Or for Node.js:
npm install openai @anthropic-ai/sdk @google/generative-ai

Common Issues / Troubleshooting

Process

Preparation

• Inventory audit: Run a provider discovery scan to confirm all expected models are accessible. Test each provider credential with a minimal request before wiring into the router.
• Define SLOs: Establish latency budgets (e.g., p95 < 5s), cost ceilings (e.g., $0.01 per request max), and reliability targets (e.g., 99.9% provider uptime before fallback).
• Select routing strategy: Choose between static routing (fixed model per task), dynamic routing (real-time cost/latency optimization), or hybrid (static defaults with dynamic fallback).
• Set up monitoring: Configure structured logging for every routing decision, provider latency percentile tracking, and cost accumulation sinks.

Execution

• Deploy the router behind your application's LLM abstraction layer. Start with a single route and one fallback model.
• Route all LLM traffic through the router. Verify routing decisions match your configuration by inspecting decision logs.
• Monitor fallback rates — if fallback exceeds 5% of traffic, investigate the primary provider's health before the circuit breaker trips.
• Stream responses through the router so fallback transitions are invisible to end users. The router should buffer intermediate state so a retry on a new model resumes from the last user message.

Stewardship

• Review cost and latency reports weekly. Rebalance routes when a cheaper model achieves comparable quality on a task.
• Update the model inventory as new provider versions release. Run shadow comparisons (route 5% traffic to the new model, measure quality) before promoting.
• Rotate API keys on a schedule. Coordinate key rotation with the router's credential store to avoid 401 errors during active traffic.
• Maintain a runbook for each provider outage scenario: expected behavior, alert thresholds, manual override procedures.

Verification

• Routes correctly by task type — all configured task-to-model mappings dispatch to the expected provider
• Fallback triggers on primary failure — inject a fake 5xx or timeout, confirm secondary model handles the request within the SLO
• Cost tracking accurate per request — compare router-reported cost against provider billing API for a sample of requests
• Latency within acceptable bounds — p50 and p95 latency per model stay under configured SLOs
• No single point of failure — all providers can be removed from routing without blocking traffic
• Circuit breaker trips and recovers — after N consecutive failures, provider is excluded; after cooldown, it resumes
• Auth failure skips immediately — 401/403 errors do not trigger retry, fall through to next model directly
• Cross-provider fallback works — simulate total outage of one provider, verify traffic redirects to a different provider entirely
• Streaming fallback preserves state — if primary fails mid-stream, the new model receives the full conversation context

Monetization