Litellm Valkey

LiteLLM proxy + Redis/Valkey multi-pod — operator reference

Target: operating LiteLLM proxy with 2+ replicas (typically the in-repo Helm chart) where Redis or Valkey is supposed to make rate limits, spend/budget enforcement, cooldowns, and locks fleet-wide. Grounded in source at 4d543245 (v1.95.0-dev, 2026-07-29; latest stable v1.94.0) plus a GitHub-issue sweep of the same date. LiteLLM releases weekly and fixes land fast — treat every claim as version-stamped, and re-verify on the deployed tag.

Sibling skill: the proxy's management REST API (keys, teams, budgets semantics) is litellm-api. Migrating the Redis itself to Valkey is redis-to-valkey.

The single most important thing to internalize: when Redis fails — or is never wired in — LiteLLM does not fail. It silently enforces everything per-pod. No 5xx, no metric, log-level warning at best. A fleet of N pods enforces N× every rate limit and lets budgets drift. Verifying that coordination is actually shared is the operator's job; nothing in the product surfaces the loss. Read references/silent-degradation.md first.

Which Redis am I actually coordinating through?

The coordination Redis (rate limits, spend counters, pod locks — NOT the response cache) is resolved in this precedence order (proxy_server.py _init_coordination_redis → _init_cache):

1. general_settings.coordination_redis in config.yaml — the only clean way to point coordination at its own Redis. First-class since v1.93.0 (PR #32661, 2026-07-11). Documented nowhere as of 2026-07-30.
2. A coordination_redis block persisted in the DB by the Admin UI — applied after the config file, so it silently outranks it. GitOps drift hazard.
3. Borrowing the response-cache client — iff litellm_settings.cache: true and the cache backend is plain Redis (RedisCache/RedisClusterCache). A semantic/disk/s3 cache is not borrowable.
4. REDIS_* env fallback — but steps 3 and 4 live inside _init_cache, which only runs on the cache: true config branch. REDIS_HOST alone, with no cache: true and no coordination_redis block, coordinates nothing — every limit is per-pod while docs/proxy/prod.md claims env Redis "shares rate limit counters across instances".

Two more traps in the same area:

• Open issue #26233: putting any key in cache_params (even mode: default_off) makes the proxy ignore REDIS_* env vars for the usage cache — silent in-memory fallback.
• Router only creates its own Redis cache when redis_url or redis_host and redis_port are set (router.py:500); redis_host without redis_port silently yields an in-memory router cache, and the router does not read env vars itself. (The proxy back-fills router.cache.redis_cache from the coordination Redis when one exists.)

Verify, don't assume: GET /coordination_redis/settings (admin) returns {"values": {...redacted config...}, "fields": [...], "source": "coordination_redis"|"cache_backend"|"environment"|null} — source names which of the four sources won; null means per-pod mode. POST /coordination_redis/settings/test pings with candidate params ({"settings": {}} tests the saved config). Both undocumented. GET /cache/ping tests only the response cache — it is blind to a separate coordination Redis. scripts/litellm-redis-preflight.sh runs the whole check.

Sentinel and Cluster — support matrix that moved late

A commonly-remembered version of this is "one function supported Sentinel, the other Cluster". The real historical split: the response cache (cache_params) supported both Sentinel and Cluster for years; the core/coordination path had no direct Sentinel config at all until v1.93.0 (and its Cluster wiring had gaps, #22748). The feature request for Sentinel in router_settings (#22796) was never implemented — the stale bot closed it 2026-06-11; coordination_redis landed by another route a month later.

Cluster mode changes semantics, not just topology: the v3 rate limiter's all-or-nothing check across key/user/team descriptors degrades to per-slot Lua calls with best-effort refunds (a crash mid-loop inflates counters until window expiry), SCAN-based paths silently return [], and the version probe fails (per-node INFO) so Redis ≤6 clusters get sent Redis-7 command forms. Details + Sentinel rough edges: references/cluster-sentinel.md.

Triage table

Per-pod vs shared — the numbers that matter

Full table in references/shared-vs-perpod.md. The staleness bounds to memorize: 1s (usage-based-routing/budget-router increments flush to Redis on DEFAULT_REDIS_SYNC_INTERVAL), 10s (DualCache batch reads throttled per key by undocumented default_redis_batch_cache_expiry — this is why routing and cooldown reads lag), 60s (in-memory tiers for budgets/auth/config; user_api_key_cache is per-pod-only unless enable_redis_auth_cache: true), 3600s (leaked parallel-request slots self-heal). The spend→DB pipeline is at-most-once: the code's own log admits "Data already popped from Redis may be lost".

Production config that actually coordinates

general_settings:
  coordination_redis:          # v1.93.0+; undocumented; wins over cache-borrow + env
    sentinel_nodes: [["valkey-sentinel-0", 26379], ["valkey-sentinel-1", 26379], ["valkey-sentinel-2", 26379]]
    service_name: "mymaster"
    password: os.environ/REDIS_PASSWORD          # data-node auth
    # sentinel_password: os.environ/SENTINEL_PW  # only if sentinels themselves require auth
  fail_closed_budget_enforcement: true   # budgets reject on Redis failure — no rate-limit equivalent exists
litellm_settings:
  cache: true                  # optional response cache; independent of coordination since v1.93.0
  cache_params: {type: redis, host: ..., port: ...}
  enable_redis_auth_cache: true  # else key revocation lags 60s per pod

Helm chart wiring (redis.sentinel.enabled renders the coordination block from masterSet; precedence rules; the Admin-UI DB copy that beats all of it): references/coordination-redis.md §Helm. After any Admin-UI use, re-check GET /coordination_redis/settings.

Gotchas that cost hours

• REDIS_CONNECTION_POOL_KWARGS (documented) is a no-op — connection_pool_kwargs is not a redis.Redis.__init__ arg so the env mapping drops it. The working knob is cache_params.max_connections.
• REDIS_SOCKET_TIMEOUT (documented, "default 0.1") is a no-op on the main client — the RedisCache.__init__ default of 5.0 always wins; in the one env path where it applies it arrives as the string "0.1".
• docs/proxy/users.md "Multi-instance rate limiting" describes the legacy v1 limiter (0.01s sync loop). The default is the v3 Lua-per-request limiter (LEGACY_MULTI_INSTANCE_RATE_LIMITING=true restores v1); no 0.01s sync exists in the tree.
• redis is not a declared dependency of the litellm package — it arrives transitively (rq, redisvl; lockfile resolves redis-py 5.3.1). SDK-only installs fail at import redis; the supported REDIS_* env-var set is introspected from the installed redis-py at import time and can shift on a transitive bump.
• POST /cache/flushall wipes coordination too when the cache client is borrowed (the common case): rate-limit counters, spend counters, spend buffer, cron locks — and any co-tenant of the instance.
• Valkey works but is invisible: Valkey-specific code exists (valkey_semantic_cache.py, ElastiCache-Valkey float-version parse fix #16207) yet grep -ri valkey docs/ finds nothing. Standard cache_params/coordination_redis against Valkey behaves as Redis 7.2; valkey-semantic explicitly rejects cluster-mode endpoints (good error, and one of the few documented bits).
• The v3 limiter's odd key shapes are load-bearing: {api_key:sk-…}:requests — the braces are cluster hash tags, not a bug. Don't "clean them up".

Task → reference routing

Scripts

• ${CLAUDE_SKILL_DIR}/scripts/litellm-redis-preflight.sh <base-url> <admin-key> — asks the proxy which coordination source won (/coordination_redis/settings), pings it (/settings/test), pings the response cache (/cache/ping), and prints the per-pod-enforcement warning if no source is active. Execution-tested against a mock proxy (source-present → exit 0, source: null → exit 2).

Non-negotiables before trusting multi-pod enforcement

• Prove coordination is wired: GET /coordination_redis/settings must name a source. Env vars alone without cache: true or a coordination_redis block = per-pod everything.
• ≥ v1.93.0 for Sentinel on the core path and for independent coordination config at all. On v1.93.0 specifically, check #34614 (redis-py 5.3.1 ssl_check_hostname) before rolling out.
• fail_closed_budget_enforcement: true if budget overrun costs real money — the default fails open, and rate limits always fail open (no closed mode exists).
• Alert on the degradation log strings (references/silent-degradation.md has the exact list) — they are the only signal that the fleet has stopped coordinating.
• One Redis logical DB per LiteLLM deployment — coordination keys are unprefixed; sharing a DB across deployments collides locks and counters.
• Load-test the actual limit: run > limit RPM across all pods and confirm 429s at ~1× (not ~N×) the configured value. This is the only end-to-end proof.