Inference Caching And Kv

Inference Caching & KV Cache Management

Three distinct caches get confused under one word. They live at different layers, have different correctness properties, and fail differently.

Rule of thumb: prompt cache is exact + cheap + safe — on always. Semantic cache is fuzzy + risky — use surgically. KV cache is not optional (it's how decode works) and at scale dominates GPU memory.

1. Prompt caching (provider-level)

How it works

The model's prefill phase computes K/V for every prompt token before the first output token, at cost O(prompt_length). Prompt caching stores that computed K/V keyed by an exact token prefix; on a hit the engine skips prefill for the cached span and resumes from the first divergent token. Matching is on exact tokens from position 0 — a prefix match, not fuzzy or substring. One changed byte invalidates everything from that byte onward. Prefill is the dominant cost for RAG/agent/long-context workloads where the prompt dwarfs the completion, so caching it turns repeated large prefixes into near-free reads.

Anthropic (explicit, cache_control breakpoints):

• Cache write ≈ 1.25× base input price (5-min TTL) or 2× (1-hr TTL).
• Cache read ≈ 0.1× base input price (~90% discount).
• Up to 4 breakpoints; each marks the end of a cacheable prefix. The engine also checks shorter prefixes, so you don't need a breakpoint per turn.
• TTL options: 5 minutes (default) and 1 hour. TTL refreshes on each hit.
• Minimum cacheable size: 1024 tokens (Sonnet/Opus) or 2048 (Haiku).

Break-even: a write costs 0.25× extra (5-min), each read saves 0.9×, so a cached prefix pays for itself after ~1 reuse within the TTL (0.25 < 0.9). A 50-call agent loop reusing a 20k-token system+tools prefix pays 1.25× once + 0.1× × 49 instead of 1.0× × 50 — roughly an ~85% input-cost cut on that prefix.

OpenAI (automatic prefix caching):

• No API flag. Prompts ≥1024 tokens are auto-cached; matched in 128-token increments (1024, 1152, 1280, …).
• Cached input tokens billed at ~0.5× (50% off, model-dependent); some newer models advertise deeper discounts. No separate write surcharge.
• TTL: evicted after 5–10 min idle, up to ~1 hr off-peak. Route the same prompt family to the same backend with a stable prompt_cache_key (or legacy user field) to raise hit rate.
• usage.prompt_tokens_details.cached_tokens reports the hit size.

Gemini:

• Implicit caching (auto, on by default for 2.5 models, ~0.25× on the cached portion, needs the common prefix at the start of the request) plus explicit caching (CachedContent — you set TTL, get a guaranteed discount + a storage fee per token-hour).

Stable-prefix design (the one thing that makes or breaks hit rate)

Order the prompt most-stable → most-volatile:

[ system instructions ]   ← never changes        ┐
[ tool / function defs ]   ← changes per deploy   │ cache this prefix
[ static context: docs,    ← changes per session  │ (breakpoint here)
  schemas, few-shot ]                             ┘
------------------------------------------------- ← cache breakpoint
[ retrieved chunks ]       ← per query   ┐ keep AFTER the breakpoint
[ conversation history ]   ← per turn    │ so the stable prefix
[ user's current message ] ← per call    ┘ survives

# Anthropic: cache the big static prefix, leave volatile tail uncached
messages = [{
  "role": "user",
  "content": [
    {"type": "text", "text": SYSTEM_AND_TOOLS},            # stable
    {"type": "text", "text": STATIC_DOCS,
     "cache_control": {"type": "ephemeral", "ttl": "1h"}}, # breakpoint
    {"type": "text", "text": user_query},                  # volatile, uncached
  ],
}]

Cache hit rate is the metric. Track: cache_read_tokens / (cache_read + cache_write + uncached_input). Healthy agent/RAG loops hit 60–90%+. A sudden drop to ~0% means the prefix went unstable (see failure modes).

What silently breaks a prefix

• A timestamp, request ID, UUID, or Date.now() injected into the system prompt.
• Reordering tools, or non-deterministic JSON key order in tool schemas.
• Trimming/whitespace differences, locale-formatted numbers, or a model/version swap (cache is per exact model).
• Putting retrieved/volatile content before the static block.
• Per-user personalization spliced into the front instead of the tail.

2. Semantic caching

Embed the incoming query → nearest-neighbour search in a vector store → if a stored entry is within a similarity threshold, return its cached response without calling the model.

def semantic_lookup(query, threshold=0.92):
    q = embed(query)
    hit, score = vector_store.nearest(q)           # cosine similarity
    if score >= threshold and not is_stale(hit):
        return hit.response                        # skip the LLM entirely
    resp = call_llm(query)
    vector_store.upsert(embed(query), resp, ts=now())
    return resp

When it helps

• High-repetition, low-variance intents: FAQ bots, support deflection, classifier prompts, doc Q&A where many users ask the same thing different ways.
• Workloads with a long tail of paraphrases of a small set of questions.

When it hurts (don't use it)

• Personalized, stateful, time-sensitive, or computational answers ("what's my balance", "current price", "summarize this doc"). Two queries that look similar can require different answers.
• Anything where a wrong-but-plausible answer is costly.

Thresholds & the false-hit risk

• Cosine 0.95+ is conservative (fewer hits, few false hits); 0.85–0.90 is aggressive (more hits, real false-hit risk). Tune on labelled pairs, not vibes — measure precision of "same intent" at each threshold.
• False hit = two semantically different queries score above threshold and the cache serves the wrong stored answer. This is the defining hazard: embeddings collapse negation ("can I cancel" vs "can I not cancel"), numbers, and entities. Mitigate: raise threshold, add a cheap LLM/rule re-rank gate on borderline hits, key on normalized intent + critical entities, never cache responses containing user-specific data.
• Staleness: cached answers go wrong when the underlying truth changes (price, policy, docs). Attach a TTL and a content-version tag; invalidate by version bump or event, not just time. Always cache with an embedding model version — re-embedding on model change is required.

Prompt-cache vs semantic-cache

They compose: semantic cache catches whole-query repeats; prompt cache makes the misses cheaper. Layer semantic in front, prompt cache underneath.

3. KV cache internals

What it is

During decode, each new token attends to all previous tokens' keys and values. Recomputing K/V for the whole sequence every step is O(n²) total. Instead the engine caches K and V per token per layer once, so each decode step only projects the one new token and attends against the cached set — O(1) in past-token work. The KV cache is what makes autoregressive decode tractable, and it grows by one token's worth of K/V every step.

Memory cost — the formula

kv_bytes = 2 · num_layers · num_kv_heads · head_dim · seq_len · batch · dtype_bytes
           │
           └─ the 2 is for K *and* V

num_kv_heads is the GQA/MQA head count (often ≪ attention heads — this is a primary KV-memory lever). dtype_bytes: FP16/BF16 = 2, FP8/INT8 = 1.

Worked example — Llama-3-8B, 32 layers, 8 KV heads (GQA), head_dim 128, FP16:

• Per token: 2 · 32 · 8 · 128 · 2 = 131,072 B ≈ 128 KiB/token.
• 8k-token sequence: 128 KiB · 8192 ≈ 1.0 GiB. Batch of 32: ~32 GiB of KV — on top of ~16 GiB of FP16 weights. On an 80 GB A100/H100 you are KV-bound long before weight-bound.

Llama-3-70B (80 layers, 8 KV heads, head_dim 128, FP16): 320 KiB/token → a single 32k-context request needs **10 GiB of KV by itself**. KV cache, not weights, sets the throughput ceiling at long context / high batch.

Takeaway: KV scales linearly with seq_len × batch; weights are fixed. Past a point, KV dominates total memory.

4. KV cache management

PagedAttention (vLLM)

Naive serving pre-allocates a contiguous KV buffer sized to max_seq_len per request → internal fragmentation and reserved-but-unused memory (often 60–80% waste). PagedAttention borrows OS virtual memory: KV is split into fixed-size blocks (e.g. 16 tokens), stored non-contiguously, with a block table mapping logical → physical. Result: near-zero fragmentation, on-demand allocation, and copy-on-write block sharing for identical prefixes (parallel samples, beam search, shared system prompts). vLLM reports up to ~2–4× throughput vs naive allocation, mostly from packing more concurrent sequences into the same VRAM.

Prefix sharing / RadixAttention (SGLang)

Caches prefixes in a radix tree keyed by token sequence, so any request sharing a prefix (system prompt, few-shot block, multi-turn history, tree-of-thought branches) reuses the computed KV instead of re-running prefill (LRU eviction on the tree). Automatic cross-request KV reuse — the server-side analogue of provider prompt caching. Huge for agent fan-out; hit rates of 50–90% are common. vLLM's --enable-prefix-caching does the same via hashed block reuse.

Eviction & scheduling

KV blocks are a finite pool. When full:

• LRU / priority eviction of cached (non-active) prefix blocks first.
• Preemption of running sequences when even active KV won't fit:
  • Recompute: drop the sequence's KV, re-run prefill later. Cheap memory, costs compute. Default for short prompts.
  • Swap/offload: move KV to CPU RAM (or disk/NVMe) and page it back. Saves recompute, costs PCIe bandwidth (offload ~tens of GB/s vs HBM ~TB/s). Better for long prompts where recompute is expensive.
• Continuous batching adds/removes sequences each step; preemption keeps the batch within the KV budget rather than OOM-crashing.

Quantized KV cache (FP8 / INT8)

Store K/V at 1 byte/elem instead of 2 → ~2× longer context or batch in the same VRAM. FP8 KV typically costs <1% quality; INT8 needs per-channel/per-token scales to stay clean. (--kv-cache-dtype fp8 in vLLM.)

Sliding-window / streaming

Models with sliding-window attention (e.g. Mistral) attend only to the last W tokens, so KV is bounded at W — O(W) memory, not O(n). StreamingLLM keeps a few "attention-sink" initial tokens + a recent window to run effectively unbounded streams without OOM (dropping the true middle) — for long-running chats/logs where old context is expendable.

KV reuse across requests

Persisting KV beyond one request (RadixAttention, vLLM prefix cache, or external stores like LMCache offloading to CPU/disk) turns repeated prefixes into reads — provider prompt caching implemented in your own serving layer when you self-host.

5. Memory pressure at scale

The core tradeoff on a fixed GPU:

VRAM ≈ weights + activations + KV(batch, context)     (KV is the variable term)

Since KV ∝ batch · context, you trade them against each other:

OOM happens when active + cached KV exceeds the pool — usually triggered by a burst of long-context requests, not steady state. Defend with: a hard max_num_seqs / max_num_batched_tokens cap, gpu_memory_utilization headroom (e.g. 0.90, not 0.98), admission control / queueing, and preemption (swap or recompute) so the server degrades to higher latency instead of crashing.

Capacity planning (back-of-envelope):

kv_budget   = (VRAM · util) − weights − activation_overhead
max_tokens  = kv_budget / bytes_per_token        # from §3 formula
max_concurrent ≈ max_tokens / avg_seq_len

Plan for p95 sequence length, not mean — long tails are what OOM you. Leave ~10–15% headroom for activation spikes and fragmentation.

6. Cache safety & multi-tenancy

Rules:

• Partition every shared cache by tenant/user when content is user-specific. A global prefix cache is safe only for genuinely shared content (the system prompt, public docs).
• Never store secrets/PII in a shared prefix that other tenants' requests could match against.
• Semantic caches must key on (tenant_id, normalized_query) and exclude any response embedding user data — or scope the vector index per tenant.

7. Failure modes (and the fix)

• Unstable prefix → ~0% hit rate. A UUID/timestamp/reordered-tool crept into the cached prefix. Symptom: cache_read_tokens collapse, input cost spikes. → Move all volatile content after the breakpoint; make tool/JSON order deterministic; diff two consecutive prompts byte-for-byte.
• Semantic false hit → wrong answer served. Threshold too low; negation or entity difference collapsed by the embedding. → Raise threshold, add a re-rank/verify gate, key on critical entities, exclude personalized intents.
• Semantic staleness → confidently outdated answer. Truth changed; cache didn't. → Version-tag + event-driven invalidation, not TTL alone.
• KV OOM under long context. A burst of long prompts exceeds the KV pool. → Cap max_num_seqs / batched_tokens, enable preemption (swap/recompute), FP8 KV, lower gpu_memory_utilization, admission-queue.
• Cross-tenant cache contamination. Shared prefix/semantic cache leaks one tenant's data to another. → Partition keys by tenant; never cache PII in a shared namespace.
• Cache thrash. Working set > cache → constant evict/refetch, worse than no cache. → Size cache to the working set, pin hot prefixes, or shorten the prefix.
• TTL expiry mid-loop. A slow agent step lets the prefix expire → surprise cache-write surcharge. → Use the 1-hr TTL for long runs; hits refresh the TTL.

8. Do / Don't

Do

• Turn on provider prompt caching for every repeated-prefix workload — it's free correctness-wise and pays off after one reuse.
• Order prompts stable→volatile; put the cache breakpoint after the static block.
• Track cache hit rate as a first-class metric; alert on a sudden drop.
• Use FP8 KV + PagedAttention + prefix caching when self-hosting at scale.
• Partition shared caches by tenant; plan KV for p95 length with headroom.

Don't

• Don't inject timestamps/UUIDs/non-deterministic ordering into a cached prefix.
• Don't semantic-cache personalized, time-sensitive, or computational answers.
• Don't set the semantic threshold by feel — measure precision on labelled pairs.
• Don't run gpu_memory_utilization at 0.98 and expect no OOM under load.
• Don't share KV blocks or cached responses across tenants for sensitive content.