Problem Addressed

TensorZero

Overview

TensorZero is an open-source stack for industrial-grade LLM applications, written in Rust for extreme performance (<1ms p99 latency). It unifies five core capabilities: a multi-provider LLM gateway, observability with feedback collection, prompt/model optimization, evaluation with LLM judges, and experimentation with A/B testing. The platform enables a data and learning flywheel where production metrics and human feedback continuously improve prompts, models, and inference strategies.

Problem Addressed

Key Statistics

Key Features

LLM Gateway

• Unified API: Access 20+ LLM providers through single interface
• Performance: <1ms p99 latency overhead at 10k+ QPS (Rust-based)
• Inference modes: Streaming, tool use, structured outputs (JSON), batch, embeddings, multimodal (images, files), caching
• Prompt templates: Type-safe schemas enforce consistent interface between application and LLMs
• High availability: Routing, retries, fallbacks, load balancing, granular timeouts
• Rate limiting: Custom rate limits with granular scopes (user-defined tags)
• Auth: Clients access models without sharing provider API keys

Supported Providers

Anthropic, AWS Bedrock, AWS SageMaker, Azure, DeepSeek, Fireworks, GCP Vertex AI (Anthropic), GCP Vertex AI (Gemini), Google AI Studio (Gemini), Groq, Hyperbolic, Mistral, OpenAI, OpenRouter, SGLang, TGI, Together AI, vLLM, xAI (Grok), and any OpenAI-compatible API (e.g., Ollama).

Observability

• Data storage: Inferences and feedback stored in your own database
• TensorZero UI: Debug individual API calls or monitor aggregate metrics
• Datasets: Build datasets for optimization, evaluation, and workflows
• Replay: Replay historical inferences with new prompts/models/strategies
• Export: OpenTelemetry traces (OTLP) and Prometheus metrics export
• Programmatic access: Query inferences with filters, compound filters, search, pagination

Optimization

• Fine-tuning: Supervised fine-tuning (SFT) and RLHF from production feedback
• Prompt engineering: Automated optimization with GEPA and MIPROv2 algorithms
• Inference strategies: Dynamic in-context learning (DICL), best-of-N sampling, mixture-of-N
• Data flywheel: Production data continuously improves models (better variants -> better data -> better variants)

Evaluation

• Inference evaluations: Evaluate individual LLM calls with heuristics or LLM judges (unit tests for LLMs)
• Workflow evaluations: Evaluate end-to-end workflows with complete flexibility (integration tests for LLMs)
• Judge optimization: Optimize LLM judges like any TensorZero function to align with human preferences
• CLI and UI: Run evaluations from command line or through the TensorZero UI

Experimentation

• Adaptive A/B testing: Multi-armed bandits for efficient variant selection
• Principled experiments: Support for multi-turn systems and sequential testing
• Confidence: Ship with statistical confidence in prompt/model changes

Technical Architecture

Stack Components

Workspace Structure

tensorzero/
├── tensorzero-core/          # Core gateway logic
├── gateway/                  # Gateway service
├── clients/
│   ├── rust/                 # Rust client
│   └── python/               # Python client (PyPI: tensorzero)
├── evaluations/              # Evaluation framework
├── tensorzero-optimizers/    # Optimization algorithms
├── provider-proxy/           # Provider proxy layer
└── internal/                 # Internal utilities
    ├── tensorzero-types/     # Type definitions
    ├── tensorzero-auth/      # Authentication
    └── autopilot-*/          # Autopilot components

Data Flow

1. Application sends inference request to TensorZero gateway
2. Gateway routes to configured provider(s) with fallback/retry logic
3. Inference logged to database with request/response data
4. Application sends feedback (metrics, human edits) via feedback API
5. Feedback linked to inference for training data collection
6. Optimization jobs consume feedback to improve prompts/models
7. New variants deployed via A/B testing framework
8. Cycle repeats: better variants -> better data -> better variants

Configuration Model

TensorZero uses TOML configuration with GitOps-friendly structure:

# tensorzero.toml
[functions.my_function]
type = "chat"

[functions.my_function.variants.gpt4o]
type = "chat_completion"
model = "gpt-4o"

[functions.my_function.variants.claude]
type = "chat_completion"
model = "claude-sonnet-4-20250514"
weight = 0.5  # A/B test allocation

Installation and Usage

Installation

# Python client
pip install tensorzero

# Using uv (recommended)
uv pip install tensorzero

Gateway Deployment

# Docker deployment
docker run -p 3000:3000 tensorzero/gateway

# With configuration
docker run -p 3000:3000 -v $(pwd)/config:/app/config tensorzero/gateway

Basic Inference (TensorZero SDK)

from tensorzero import TensorZeroGateway  # or AsyncTensorZeroGateway

with TensorZeroGateway.build_embedded(...) as t0:
    response = t0.inference(
        model_name="openai::gpt-4o-mini",
        # Switch providers easily: "anthropic::claude-sonnet-4-5"
        input={
            "messages": [
                {"role": "user", "content": "Write a haiku about TensorZero."}
            ]
        },
    )

Using with OpenAI SDK

from openai import OpenAI
from tensorzero import patch_openai_client

client = OpenAI()
patch_openai_client(client, ...)

response = client.chat.completions.create(
    model="tensorzero::model_name::openai::gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}],
)

Feedback Collection

# Record feedback for optimization
t0.feedback(
    inference_id=response.inference_id,
    metric_name="user_rating",
    value=5,
)

Running Evaluations (CLI)

docker compose run --rm evaluations \
  --evaluation-name extract_data \
  --dataset-name hard_test_cases \
  --variant-name gpt_4o \
  --concurrency 5

Example Use Cases

The repository includes complete runnable examples:

Relevance to Claude Code Development

Direct Applications

1. LLM Gateway Patterns: Unified API with fallbacks, retries, and rate limiting provides reference architecture for resilient LLM integrations.

2. Feedback Loop Design: Metrics and human feedback collection enabling continuous improvement could inform Claude Code skill refinement workflows.

3. Evaluation Framework: Inference-level and workflow-level evaluations with LLM judges offer testing patterns for skill quality assurance.

4. Observability Patterns: Structured logging with programmatic query access provides model for tracking Claude Code operations.

5. Configuration-as-Code: TOML-based prompt templates and variant management demonstrate GitOps approach for LLM configurations.

Patterns Worth Adopting

1. Dynamic In-Context Learning (DICL): Automatically selecting relevant examples from historical data improves LLM performance without fine-tuning.

2. Adaptive A/B Testing: Multi-armed bandits efficiently identify best prompts/models with statistical confidence.

3. Type-Safe LLM Interfaces: Schema enforcement on prompts creates contracts between application and LLM.

4. Feedback-to-Training Pipeline: Direct path from production feedback to fine-tuning enables continuous improvement.

5. Evaluation Categories: Separating inference evaluations (unit tests) from workflow evaluations (integration tests) provides clear testing taxonomy.

Integration Opportunities

1. Gateway for Multi-Provider Access: TensorZero could serve as backend for Claude Code operations requiring multiple LLM providers.

2. Feedback Collection: Integrate TensorZero feedback API to collect metrics on Claude Code skill effectiveness.

3. Prompt Optimization: Use GEPA/MIPROv2 algorithms to optimize Claude Code prompts and instructions.

4. Evaluation Infrastructure: Leverage evaluation framework to test skill quality with LLM judges.

5. Observability Export: Export Claude Code operations to TensorZero for unified LLM observability.

Comparison with Claude Code

References

Research Method: Information gathered from official GitHub repository README, GitHub API (stars, forks, issues, contributors, releases), PyPI package metadata, PyPI download statistics API, and Cargo.toml workspace configuration. Statistics verified via direct API calls on 2026-01-31.

Freshness Tracking

Review Triggers:

• Major version release
• Significant new optimization algorithm
• GitHub stars milestone (15K, 20K)
• PyPI downloads milestone (100K monthly)
• TensorZero Autopilot general availability
• New model provider integrations of note
• Breaking changes to gateway API or configuration format
• Significant updates to evaluation framework