2817 Proxy Integration Plan Da078dae

Proxy Integration Plan

Goals

• Provide a single proxy service that acts as a drop-in replacement for both OpenAI and Anthropic APIs.
• Support OpenAI /v1/chat/completions (route GPT-4o-mini → GPT-4o cascade).
• Support Anthropic /v1/messages (route Haiku → Sonnet cascade).
• Preserve client compatibility with official SDKs, LangChain, and Claude Code.
• Maintain shared cascade logic, consistent telemetry, and per-provider cost tracking.

Reference MVP

• Existing MVP example: examples/fastapi_proxy_routing.py (Anthropic-only). Extend it to support OpenAI format while sharing cascade logic.

Dual-Provider Architecture (Single Proxy)

• One FastAPI service with two router groups:
  • /v1/chat/completions → OpenAI-compatible handler.
  • /v1/messages → Anthropic-compatible handler.
• Shared core pipeline:
  • Request normalization → cascade selection → provider dispatch → response normalization.
• Provider adapters implement the format translation, model mapping, and response shaping per standard.

Endpoint Mapping

OpenAI-Compatible

• Endpoint: POST /v1/chat/completions
• Request mapping:
  • model: map GPT-4o-mini → GPT-4o cascade default.
  • messages: convert to internal message schema (role/content).
  • temperature, max_tokens, top_p, stream, etc.: pass through or apply defaults.
• Response mapping:
  • Convert internal completion to choices[0].message with role=assistant.
  • Populate usage and id per OpenAI response format.

Anthropic-Compatible

• Endpoint: POST /v1/messages
• Request mapping:
  • model: map Haiku → Sonnet cascade default.
  • messages: convert to internal message schema (role/content).
  • max_tokens, temperature, top_p, stream, etc.: pass through or apply defaults.
• Response mapping:
  • Convert internal completion to Anthropic content array.
  • Populate usage and id per Anthropic response format.

Response Format Translation

• Internal canonical response:
  • content: assistant text
  • tool_calls: if supported
  • usage: prompt/completion/total tokens
  • model: resolved cascade model
  • request_id: proxy-generated ID
• OpenAI output:
  • choices = [{ index, message: { role: "assistant", content }, finish_reason }]
  • usage = { prompt_tokens, completion_tokens, total_tokens }
• Anthropic output:
  • content = [{ type: "text", text }]
  • usage = { input_tokens, output_tokens }

Shared Cascade Logic

• Central cascade policy used by both providers:
  • Step 1: fast/cheap model (GPT-4o-mini or Haiku).
  • Step 2: fallback to higher-quality model (GPT-4o or Sonnet) when confidence/quality checks fail.
• Confidence checks via existing scoring/validation hooks.
• Unified telemetry: log cascade steps, latency, and model decisions with provider tags.

Configuration (Default Models Per Provider)

• Env-driven defaults:
  • OPENAI_DEFAULT_MODEL=gpt-4o-mini
  • OPENAI_CASCADE_FALLBACK=gpt-4o
  • ANTHROPIC_DEFAULT_MODEL=claude-3-haiku
  • ANTHROPIC_CASCADE_FALLBACK=claude-3-sonnet
• Allow overrides per request, but enforce provider mapping logic.

Cost Tracking Per Provider

• Tag costs by provider and resolved model.
• Aggregate metrics:
  • provider=openai|anthropic
  • model=<resolved_model>
  • cascade_step=<primary|fallback>
• Emit metrics to existing cost tracking interfaces and examples.

Implementation Steps

1. Create shared request/response normalization layer.
2. Add OpenAI router and adapter for /v1/chat/completions.
3. Extend Anthropic handler to use shared cascade pipeline.
4. Add response translation for both standards.
5. Add provider-aware cost tracking and logging.
6. Update examples to include OpenAI-compatible proxy usage.

Virtual Model Names

The proxy exposes virtual models that abstract cascade behavior:

Usage

# Client just specifies virtual model
client = OpenAI(base_url="http://localhost:8000")
response = client.chat.completions.create(
    model="cascadeflow-auto",  # Proxy handles the cascade
    messages=[{"role": "user", "content": "Hello"}]
)

Model Mapping

When proxy receives a virtual model:

1. Parse cascade strategy from model name
2. Select appropriate drafter/verifier pair
3. Apply strategy-specific thresholds
4. Return response with actual model used in metadata

Compatibility Notes

• OpenAI SDK: base URL override points to proxy.
• LangChain: use OpenAI-compatible route with environment variables.
• Anthropic SDK / Claude Code: base URL override points to proxy.
• Virtual models: work with any OpenAI-compatible client.