Setting Up a Fake OpenAI Endpoint

import Image from '@theme/IdealImage';

Benchmarks

Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.

Setting Up a Fake OpenAI Endpoint

For load testing and benchmarking, you can use a fake OpenAI proxy server. LiteLLM provides:

1. Hosted endpoint: Use our free hosted fake endpoint at https://exampleopenaiendpoint-production.up.railway.app/
2. Self-hosted: Set up your own fake OpenAI proxy server using github.com/BerriAI/example_openai_endpoint

Use this config for testing:

model_list:
  - model_name: "fake-openai-endpoint"
    litellm_params:
      model: openai/any
      api_base: https://exampleopenaiendpoint-production.up.railway.app/  # or your self-hosted endpoint
      api_key: "test"

2 Instance LiteLLM Proxy

In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.

Performance Metrics

4 Instances

Key Findings

• Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
• High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
• Setting workers equal to CPU count gives optimal performance.

/realtime API Benchmarks

End-to-end latency benchmarks for the /realtime endpoint tested against a fake realtime endpoint.

Performance Metrics

Test Setup

Machine Spec used for testing

Each machine deploying LiteLLM had the following specs:

• 4 CPU
• 8GB RAM

Configuration

• Database: PostgreSQL
• Redis: Not used

Infrastructure Recommendations

Recommended specifications based on benchmark results and industry standards for API gateway deployments.

PostgreSQL

Required for authentication, key management, and usage tracking.

Configuration: Set proxy_batch_write_at: 60 to batch writes and reduce DB load. Total connections = pool limit × instances.

Redis (Recommended)

Redis was not used in these benchmarks but provides significant production benefits: 60-80% reduced DB load.

Requirements: Redis 7.0+, AOF persistence enabled, allkeys-lru eviction policy.

Configuration:

router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

litellm_settings:
  cache: True
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD

:::tip Use redis_host, redis_port, and redis_password instead of redis_url for ~80 RPS better performance. :::

Scaling: DB connections scale linearly with instances. Consider PostgreSQL read replicas beyond 5K RPS.

See Production Configuration for detailed best practices.

Locust Settings

• 1000 Users
• 500 user Ramp Up

How to measure LiteLLM Overhead

All responses from litellm will include the x-litellm-overhead-duration-ms header, this is the latency overhead in milliseconds added by LiteLLM Proxy.

If you want to measure this on locust you can use the following code:

import os
import uuid
from locust import HttpUser, task, between, events

# Custom metric to track LiteLLM overhead duration
overhead_durations = []

@events.request.add_listener
def on_request(request_type, name, response_time, response_length, response, context, exception, start_time, url, **kwargs):
    if response and hasattr(response, 'headers'):
        overhead_duration = response.headers.get('x-litellm-overhead-duration-ms')
        if overhead_duration:
            try:
                duration_ms = float(overhead_duration)
                overhead_durations.append(duration_ms)
                # Report as custom metric
                events.request.fire(
                    request_type="Custom",
                    name="LiteLLM Overhead Duration (ms)",
                    response_time=duration_ms,
                    response_length=0,
                )
            except (ValueError, TypeError):
                pass

class MyUser(HttpUser):
    wait_time = between(0.5, 1)  # Random wait time between requests

    def on_start(self):
        self.api_key = os.getenv('API_KEY', 'sk-1234567890')
        self.client.headers.update({'Authorization': f'Bearer {self.api_key}'})

    @task
    def litellm_completion(self):
        # no cache hits with this
        payload = {
            "model": "db-openai-endpoint",
            "messages": [{"role": "user", "content": f"{uuid.uuid4()} This is a test there will be no cache hits and we'll fill up the context" * 150}],
            "user": "my-new-end-user-1"
        }
        response = self.client.post("chat/completions", json=payload)
        
        if response.status_code != 200:
            # log the errors in error.txt
            with open("error.txt", "a") as error_log:
                error_log.write(response.text + "\n")

LiteLLM vs Portkey Performance Comparison

Test Configuration: 4 CPUs, 8 GB RAM per instance | Load: 1k concurrent users, 500 ramp-up Versions: Portkey v1.14.0 | LiteLLM v1.79.1-stable
Test Duration: 5 minutes

Multi-Instance (4×) Performance

Lower is better for latency metrics; higher is better for requests and RPS.

Technical Insights

Portkey

Pros

• Low memory footprint
• Stable latency with minimal spikes

Cons

• CPU utilization capped around ~40%, indicating underutilization of available compute resources
• Experienced three I/O timeout outages

LiteLLM

Pros

• Fully utilizes available CPU capacity
• Strong connection handling and low latency after initial warm-up spikes

Cons

• High memory usage during initialization and per request

Logging Callbacks

GCS Bucket Logging

Using GCS Bucket has no impact on latency, RPS compared to Basic Litellm Proxy

LangSmith logging

Using LangSmith has no impact on latency, RPS compared to Basic Litellm Proxy