Litellm

LiteLLM AI Gateway Operations

Use this skill to operate LiteLLM as an organization's AI gateway: run the proxy (litellm --config config.yaml), route requests to 100+ LLM providers through one OpenAI-compatible API, manage model lists, routing and reliability, virtual keys, teams, budgets and rate limits, caching, guardrails, observability, and spend — and diagnose failures with evidence. LiteLLM ships two surfaces: a Python SDK (litellm.completion(), in-process) and the proxy (a FastAPI service on port 4000 with keys, budgets, and an admin UI). This is a tool skill for the named tool. Engine selection and serving methodology belong to ml-engineering; operating a single engine belongs to vllm or llama-cpp.

Operating contract

1. Record the deployment before tuning it. Capture the pinned image or pip version, config.yaml, model list, routing, budgets, env-var references, and data stores in the proxy config record. That record is the rollback unit.
2. Confirm the target, scope, and rollback path before mutating. Read-only discovery (health probes, /v1/models, logs, spend queries) may proceed without confirmation. Mutations — config changes, key mint/revocation, restarts, image upgrades, DB migrations — require an explicit human directive naming the deployment.
3. A proxy that responds is not a proxy that serves. /health/liveliness returning 200 proves liveness only. Verify at the delivery boundary: a representative /v1/chat/completions request returns tokens and x-litellm-model-id names the deployment you expected.
4. Keep evidence bounded. Summarize logs and configs; never dump full logs, .env contents, master keys, or provider credentials into chat. Spend logs and debug output can contain prompt content — redact before sharing.
5. Pin versions. LiteLLM releases weekly and changes defaults; every claim here was checked against 1.97.0 (2026-08-22). Re-verify version-sensitive behavior against your installed release before relying on it.

The litellm-health script

scripts/litellm-health is a read-only probe for a running proxy. It issues GET requests only, never writes files, and emits bounded output.

scripts/litellm-health --help                                   # no proxy needed
scripts/litellm-health --url http://127.0.0.1:4000 --json
scripts/litellm-health --check health --check readiness --json
scripts/litellm-health --check models --check model_info \
  --key "$LITELLM_MASTER_KEY" --json

Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Checks: health (GET /health/liveliness, unauthenticated), readiness (GET /health/readiness, unauthenticated; 503 when the configured DB is unreachable), models (GET /v1/models, requires key), and model_info (GET /model/info, requires key). Keys are sent as Authorization: Bearer <key>. The script never sends data anywhere except the proxy you name.

Operating loop

1. Identify the deployment: pinned version/image digest, how it runs (bare, Docker, Compose, Helm), config source (file, store_model_in_db, or both), and data stores (Postgres? Redis?).
2. Collect evidence: litellm-health --json; GET /v1/models and /model/info with a key; response headers (x-litellm-call-id, x-litellm-model-id, x-litellm-model-api-base, x-litellm-version); --detailed_debug logs or LITELLM_LOG=DEBUG for the outbound request.
3. Triage against the symptom: classify provider vs gateway errors (see troubleshooting); check cooldown state, budgets, DB connectivity.
4. Act with confirmation: bounded, scoped changes after a human directive, with the rollback path named first.
5. Verify: re-run the probe and a representative chat request at the delivery boundary.

Quickstart: one config, many providers

model_list:
  - model_name: gpt-4o                     # name clients request
    litellm_params:
      model: openai/gpt-4o                 # routed string (provider prefix required)
      api_key: os.environ/OPENAI_API_KEY   # resolved inside the proxy process
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY   # require auth on every call

Start with litellm --config config.yaml --port 4000. Success logs Proxy initialized with Config, Set models:. Clients call the OpenAI surface: /v1/chat/completions, /chat/completions, /v1/embeddings, /v1/images/generations, /v1/audio/transcriptions, plus /responses, Anthropic-compatible /messages, /model/info, /health/liveliness, /health/readiness. Any OpenAI SDK works unchanged: openai.OpenAI(base_url="http://localhost:4000", api_key=<virtual key>). Details and the SDK surface: quickstart reference.

Config and routing

• Entries sharing a model_name form one load-balanced group; each entry is a deployment with its own hashed model_id used for health and cooldown tracking.
• router_settings.routing_strategy — simple-shuffle (default, recommended; weighted by rpm/tpm or weight under litellm_params), least-busy, latency-based-routing, usage-based-routing (docs warn against it in prod), cost-based-routing.
• Reliability: litellm_settings.num_retries (per-deployment and request-level overrides exist; num_retries is not the provider SDK's max_retries), fallbacks / context_window_fallbacks / content_policy_fallbacks, cooldowns (allowed_fails, cooldown_time), deployment order for priority, enable_pre_call_checks: true to enforce context windows and region filters pre-call (opt-in).
• With store_model_in_db: true, UI/API writes deep-merge over YAML in Postgres and win on key conflicts — editing those YAML keys later has no effect while the DB row exists. Details: config and routing reference.

Keys, teams, budgets, spend

• general_settings.master_key (must start sk-) is the admin credential and UI password. Virtual keys (POST /key/generate) scope models, budgets, and rpm/tpm per workload; keys are stored hashed and never contain provider credentials.
• Budgets require Postgres. Without a connected DB, budgets fail open (a startup warning is the only signal) and key endpoints return No connected db. — never run a budget-sensitive deployment DB-less.
• Team keys enforce team (+ team-member) budgets only; the owner's personal budget does not apply. Rate limits do not apply to proxy admins. Spend lands in /spend/logs and /global/spend; store_prompts_in_spend_logs defaults to false. Details: keys and budgets reference.

Caching and guardrails

• Response cache: litellm_settings.cache: true + cache_params.type: redis for multi-instance production (in-memory is per-process; disk/S3/GCS exist). Per-request controls: cache: {ttl, no-cache, namespace} in the body.
• Semantic caches (qdrant-semantic, redis-semantic, valkey-semantic) embed the whole messages array and can replay stale answers across similar multi-turn turns — docs recommend excluding agentic traffic from semantic caching.
• Guardrails run pre_call, post_call, during_call, or logging_only (there is no all mode); Presidio PII masking is OSS. Violations fail with HTTP 400 and an embedded verdict; x-litellm-applied-guardrails names what ran. Details: caching and guardrails reference.

Observability and logging

• Callbacks: litellm_settings.success_callback / failure_callback / callbacks (Langfuse, OTel, Prometheus, Datadog, Sentry, ...). Prometheus /metrics requires auth since 1.85.0 — give the scraper a bearer key or set require_auth_for_metrics_endpoint: false.
• Forensic response headers: x-litellm-call-id, x-litellm-model-id, x-litellm-model-api-base, x-litellm-version, x-litellm-response-cost.
• Privacy: turn_off_message_logging: true keeps metadata but drops content from callbacks; redact_user_api_key_info: true redacts key/user/team identifiers. Debug with --detailed_debug, LITELLM_LOG=DEBUG, or per-request "litellm_request_debug": true. Details: observability reference.

Deployment

• Postgres is mandatory for keys, teams, spend, budgets, and UI state; Redis >=7 is required for more than one instance (shared rate-limit counters, cooldowns, cache).
• Pin image tags (ghcr.io/berriai/litellm:vX.Y.Z — semver tags since 1.84.0; -stable suffixes are gone, main-latest is deprecated). Images are cosign-signed.
• Prisma migrations run at startup by default; on Kubernetes use the migration job pattern with DISABLE_SCHEMA_UPDATE=true on serving pods. One Uvicorn worker per pod; size the DB pool as MAX_DB_CONNECTIONS / (instances x workers). Details: deployment reference.

Security and public hosting

• Version floor for any internet-reachable proxy: >=1.83.7 (CVE-2026-42208 pre-auth SQLi, CVE-2026-42203 SSTI, CVE-2026-42271 command injection, plus Starlette >=1.0.1 for the CVE-2026-48710 host-header chain). Two of these were CISA KEV-listed and actively exploited in 2026.
• Never expose management routes (/key/*, /user/*, /team/*, /config/*, /model/*, /spend/*, /ui, /prompts/test, /mcp-rest/*). Route lockdown via allowed_routes is Enterprise — on OSS, enforce at the reverse proxy.
• LITELLM_SALT_KEY encrypts DB-stored provider credentials; set it once and never rotate it after adding models. Rotate the master key only via the documented flow.
• March 2026 supply-chain incident: backdoored litellm==1.82.7/.8 PyPI wheels (~40 minutes). Prefer cosign-verified pinned images over unpinned pip installs. Hardening checklist: security reference.

Troubleshooting: the master diagnostic rule

If the error contains <Provider>Exception, the provider failed — not the gateway. AnthropicException, OpenAIException, BedrockException, ... mean the upstream call happened and its response is the evidence. No provider name means the gateway itself rejected the call (bad LiteLLM key, unknown model, cooldowns, budget).

Full taxonomy and fixes: troubleshooting reference.

Reference routing

Included artifacts

• scripts/litellm-health: read-only proxy probe (stdlib-only, --json, --check subsets, --key for authenticated routes, --help without a server).
• tests/test_litellm_health.py: deterministic tests against a local stub HTTP server, including the read-only contract.
• templates/proxy-config-record.md and templates/proxy-deployment.md: fillable records — the config record is the rollback unit; the deployment record freezes the runtime (image digest, ports, env, data stores, probes, rollback).
• references/: nine dated, source-indexed references covering the topics above.
• evals/evals.json: six output-quality evaluation cases.

Verification boundary

Hard boundaries

• Never mutate a production proxy (config, keys, teams, budgets, image, DB) without an explicit human directive naming the target and a stated rollback path. Read-only discovery may proceed freely.
• Never expose the master key, management routes, or /ui beyond the trust boundary; authentication is not a substitute for network and TLS controls.
• Never commit provider keys, DATABASE_URL, LITELLM_MASTER_KEY, or LITELLM_SALT_KEY anywhere; use os.environ/ references and a secret manager.
• Never run a budget-sensitive public deployment without Postgres — budgets fail open without one.
• Never treat a 200 from /health/liveliness as proof the gateway serves; verify at the delivery boundary.

When not to use

• Engine selection, serving methodology, quantization decisions, evaluation design — that is ml-engineering.
• Operating a single inference engine — vllm for vLLM, llama-cpp for the llama.cpp stack. LiteLLM routes to engines; it does not replace their own operation.
• Kubernetes/Docker fundamentals and reverse-proxy/TLS configuration — that is kubernetes, docker-compose, and traefik; this skill covers the LiteLLM-specific layer.
• Building applications on top of an LLM API (app architecture, agent frameworks) — that is backend/frontend engineering; this skill owns the gateway and its SDK.