Amey-Thakur Prompt caching
Providers can cache a stable prefix of the prompt and skip recomputing it, cutting input cost sharply and shrinking time-to-first-token. The win is real but conditional: it only applies to the exact repeated prefix, so prompts must be structured to keep that prefix stable.
Method
- Put the stable content first, the variable content last. System prompt, tool definitions, and few-shot examples at the top (they rarely change); the user's turn and per-request data at the bottom. The cache matches from the start, so a single early change (a timestamp, a shuffled example) invalidates everything after it (see context-engineering's stable-prefix rule).
- Freeze the prefix byte-for-byte. Caching keys on exact content: avoid injecting the current time, request IDs, randomized example order, or reformatted whitespace into the cached region. What varies per call belongs after the cache boundary, not woven through it.
- Know your provider's mechanism and thresholds. Providers differ on whether caching is automatic or explicitly marked, the minimum cacheable size, the cache lifetime (often minutes), and the discount. Read the current contract for your platform rather than assuming (your provider's documentation is the authority).
- Design multi-turn and agent loops to reuse the prefix. In a conversation or an agent loop (see agentic-loops), the growing history after the fixed prefix still lets the prefix hit; keep the system layer stable across turns so every call reuses it. Frequent system-prompt edits forfeit the cache on every turn.
- Verify hit rates, do not assume them. Instrument cached-token counts in telemetry (see llm-observability); a prompt you believe is cached but is not (a stray variable in the prefix) pays full price silently. Measure before and after structuring for caching.
- Fold caching into the cost model. Cached input is much cheaper than fresh input but not free, and output tokens are unaffected: caching attacks the input and latency side, and combines with model tiering, prompt diet, and batching (see llm-cost-latency for the full toolkit).
Boundaries
- Caching reduces cost and latency; it changes nothing about output quality or correctness, and must never be a reason to bloat the stable prefix with content the task does not need (see context-engineering).
- Cache lifetimes are short: bursty repeated traffic benefits most; widely-spaced calls may never hit, so do not over-engineer for it.
- Semantic caching (returning a stored answer for a similar question) is a different technique with its own freshness and correctness risks; do not conflate it with prefix caching (see llm-cost-latency).