Prometheus metrics

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import Image from '@theme/IdealImage';

📈 Prometheus metrics

LiteLLM Exposes a /metrics endpoint for Prometheus to Poll

Quick Start

If you're using the LiteLLM CLI with litellm --config proxy_config.yaml then you need to pip install prometheus_client==0.20.0. This is already pre-installed on the litellm Docker image

Add this to your proxy config.yaml

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
litellm_settings:
  callbacks:
    - prometheus

Start the proxy

litellm --config config.yaml --debug

Test Request

curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Content-Type: application/json' \
    --data '{
    "model": "gpt-4o",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
}'

View Metrics on /metrics, Visit http://localhost:4000/metrics

http://localhost:4000/metrics

# <proxy_base_url>/metrics

Multiple Workers

When using LiteLLM with multiple workers, you need to set the PROMETHEUS_MULTIPROC_DIR environment variable to enable aggregated metric collection across worker processes.

export PROMETHEUS_MULTIPROC_DIR="/prometheus_multiproc"

This directory is used by the Prometheus client library to store metric files that can be shared across multiple worker processes. Make sure the directory exists and is writable by your LiteLLM process.

Virtual Keys, Teams, Internal Users

Use this for for tracking per user, key, team, etc.

Team - Budget

Virtual Key - Budget

Virtual Key - Rate Limit

Initialize Budget Metrics on Startup

If you want litellm to emit the budget metrics for all keys, teams irrespective of whether they are getting requests or not, set prometheus_initialize_budget_metrics to true in the config.yaml

How this works:

• If the prometheus_initialize_budget_metrics is set to true
  • Every 5 minutes litellm runs a cron job to read all keys, teams from the database
  • It then emits the budget metrics for each key, team
  • This is used to populate the budget metrics on the /metrics endpoint

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_initialize_budget_metrics: true

Pod Health Metrics

Use these to measure per-pod queue depth and diagnose latency that occurs before LiteLLM starts processing a request.

When to use this

LiteLLM measures latency from when its handler starts. If a request waits in uvicorn's event loop before the handler runs, that wait is invisible to LiteLLM's own logs. litellm_in_flight_requests shows how loaded the pod was at any point in time.

high in_flight_requests + high ALB TargetResponseTime → pod overloaded, scale out
low  in_flight_requests + high ALB TargetResponseTime → delay is pre-ASGI (event loop blocking)

You can also check the current value directly without Prometheus:

curl http://localhost:4000/health/backlog \
  -H "Authorization: Bearer sk-..."
# {"in_flight_requests": 47}

Proxy Level Tracking Metrics

Use this to track overall LiteLLM Proxy usage.

• Track Actual traffic rate to proxy
• Number of client side requests and failures for requests made to proxy

Callback Logging Metrics

Monitor failures while shipping logs to downstream callbacks like s3_v3 cold storage

Supported Callbacks:

• S3Logger - S3 v2 cold storage failures
• langfuse - Langfuse logging failures
• otel - OpenTelemetry logging failures

LLM Provider Metrics

Use this for LLM API Error monitoring and tracking remaining rate limits and token limits

Labels Tracked

Success and Failure

Remaining Requests and Tokens

Deployment State

Fallback (Failover) Metrics

Request Counting Metrics

Request Latency Metrics

Tracking end_user on Prometheus

By default LiteLLM does not track end_user on Prometheus. This is done to reduce the cardinality of the metrics from LiteLLM Proxy.

If you want to track end_user on Prometheus, you can do the following:

litellm_settings:
  callbacks: ["prometheus"]
  enable_end_user_cost_tracking_prometheus_only: true

Emit Stream Label

Add a stream label to litellm_proxy_total_requests_metric to split requests by streaming vs. non-streaming. Disabled by default.

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_emit_stream_label: true

When enabled, litellm_proxy_total_requests_metric gains a stream label with values "True", "False", or "None".

litellm_proxy_total_requests_metric{..., stream="True"} 42
litellm_proxy_total_requests_metric{..., stream="False"} 100

:::note This label is opt-in because adding a new label to an existing metric changes its cardinality and breaks existing Prometheus queries / Grafana dashboards that target this metric. Enable it only on fresh deployments or when you are ready to update your dashboards. :::

[BETA] Custom Metrics

Track custom metrics on prometheus on all events mentioned above.

Custom Metadata Labels

1. Define the custom metadata labels in the config.yaml

model_list:
  - model_name: openai/gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_metadata_labels: ["metadata.foo", "metadata.bar"]

2. Make a request with the custom metadata labels

curl -L -X POST 'http://0.0.0.0:4000/key/generate' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
    "metadata": {
        "foo": "hello world"
    }
}'

curl -L -X POST 'http://0.0.0.0:4000/team/new' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
    "metadata": {
        "foo": "hello world"
    }
}'

3. Check your /metrics endpoint for the custom metrics

... "metadata_foo": "hello world" ...

Custom Tags

Track specific tags as prometheus labels for better filtering and monitoring.

1. Define the custom tags in the config.yaml

model_list:
  - model_name: openai/gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_metadata_labels: ["metadata.foo", "metadata.bar"]
  custom_prometheus_tags: 
    - "prod"
    - "staging"
    - "batch-job"
    - "User-Agent: RooCode/*"
    - "User-Agent: claude-cli/*"

2. Make a request with tags

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <LITELLM_API_KEY>' \
-d '{
    "model": "openai/gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          }
        ]
      }
    ],
    "max_tokens": 300,
    "metadata": {
        "tags": ["prod", "user-facing"]
    }
}'

3. Check your /metrics endpoint for the custom tag metrics

... "tag_prod": "true", "tag_staging": "false", "tag_batch_job": "false" ...

How Custom Tags Work:

• Each configured tag becomes a boolean label in prometheus metrics
• If a tag matches (exact or wildcard), the label value is "true", otherwise "false"
• Tag names are sanitized for prometheus compatibility (e.g., "batch-job" becomes "tag_batch_job")
• Wildcard patterns supported using * (e.g., "User-Agent: RooCode/*" matches "User-Agent: RooCode/1.0.0")

Example with wildcards:

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_tags:
    - "User-Agent: RooCode/*"
    - "User-Agent: claude-cli/*"

Use Cases:

• Environment tracking (prod, staging, dev)
• Request type classification (batch-job, user-facing, background)
• Feature flags (new-feature, beta-users)
• Team or service identification (team-a, service-xyz)
• User-Agent Tracking - use this to track how much Roo Code, Claude Code, Gemini CLI are used (User-Agent: RooCode/*, User-Agent: claude-cli/*, User-Agent: gemini-cli/*)

Configuring Metrics and Labels

You can selectively enable specific metrics and control which labels are included to optimize performance and reduce cardinality.

Enable Specific Metrics and Labels

Configure which metrics to emit by specifying them in prometheus_metrics_config. Each configuration group needs a group name (for organization) and a list of metrics to enable. You can optionally include a list of include_labels to filter the labels for the metrics.

model_list:
 - model_name: gpt-4o
    litellm_params:
      model: gpt-4o

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_metrics_config:
    # High-cardinality metrics with minimal labels
    - group: "proxy_metrics"
      metrics:
        - "litellm_proxy_total_requests_metric"
        - "litellm_proxy_failed_requests_metric"
      include_labels:
        - "hashed_api_key"
        - "requested_model"
        - "model_group"

On starting up LiteLLM if your metrics were correctly configured, you should see the following on your container logs

<Image img={require('../../img/prom_config.png')} style={{width: '100%', display: 'block', margin: '2rem auto'}} />

Filter Labels Per Metric

Control which labels are included for each metric to reduce cardinality:

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_metrics_config:
    - group: "token_consumption"
      metrics:
        - "litellm_input_tokens_metric"
        - "litellm_output_tokens_metric"
        - "litellm_total_tokens_metric"
      include_labels:
        - "model"
        - "team"
        - "hashed_api_key"
    - group: "request_tracking"
      metrics:
        - "litellm_proxy_total_requests_metric"
      include_labels:
        - "status_code"
        - "requested_model"

Advanced Configuration

You can create multiple configuration groups with different label sets:

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_metrics_config:
    # High-cardinality metrics with minimal labels
    - group: "deployment_health"
      metrics:
        - "litellm_deployment_success_responses"
        - "litellm_deployment_failure_responses"
      include_labels:
        - "api_provider"
        - "requested_model"
    
    # Budget metrics with full label set
    - group: "budget_tracking"
      metrics:
        - "litellm_remaining_team_budget_metric"
      include_labels:
        - "team"
        - "team_alias"
        - "hashed_api_key"
        - "api_key_alias"
        - "model"
        - "end_user"
    
    # Latency metrics with performance-focused labels
    - group: "performance"
      metrics:
        - "litellm_request_total_latency_metric"
        - "litellm_llm_api_latency_metric"
      include_labels:
        - "model"
        - "api_provider"
        - "requested_model"

Configuration Structure:

• group: A descriptive name for organizing related metrics
• metrics: List of metric names to include in this group
• include_labels: (Optional) List of labels to include for these metrics

Default Behavior: If no prometheus_metrics_config is specified, all metrics are enabled with their default labels (backward compatible).

Monitor System Health

To monitor the health of litellm adjacent services (redis / postgres), do:

model_list:
 - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
litellm_settings:
  service_callback: ["prometheus_system"]

DB Transaction Queue Health Metrics

Use these metrics to monitor the health of the DB Transaction Queue. Eg. Monitoring the size of the in-memory and redis buffers.

🔥 LiteLLM Maintained Grafana Dashboards

Link to Grafana Dashboards maintained by LiteLLM

https://github.com/BerriAI/litellm/tree/main/cookbook/litellm_proxy_server/grafana_dashboard

Here is a screenshot of the metrics you can monitor with the LiteLLM Grafana Dashboard

<Image img={require('../../img/grafana_1.png')} />

<Image img={require('../../img/grafana_2.png')} />

<Image img={require('../../img/grafana_3.png')} />

Deprecated Metrics

Add authentication on /metrics endpoint

By default /metrics endpoint is unauthenticated.

You can opt into running litellm authentication on the /metrics endpoint by setting the following on the config

litellm_settings:
  require_auth_for_metrics_endpoint: true

FAQ

What are _created vs. _total metrics?

• _created metrics are metrics that are created when the proxy starts
• _total metrics are metrics that are incremented for each request

You should consume the _total metrics for your counting purposes