Inference Caching And Kv

Reference-grade guide to caching in LLM inference — provider prompt caching (Anthropic cache_control breakpoints, OpenAI automatic prefix caching, Gemini implicit/explicit), semantic caching, and KV-cache internals & management (PagedAttention/vLLM, RadixAttention/SGLang, eviction, quantized KV, memory pressure, multi-tenant safety). Concrete numbers, formulas, failure modes.

jpoindexter Updated 2 repo stars

File contents

jpoindexter/design-and-ai-skills commit cf5d78a7a1

Frequently asked questions

npx get-skill add jpoindexter/inference-caching-and-kv