Caching

Caching

A cache is a second copy of the truth that is allowed to be wrong. Adding one trades a latency problem for a correctness problem, and the correctness problem arrives later, in production, as "why is this user seeing yesterday's number".

Never add a cache before you can say, in one sentence each: what is cached, keyed by what, for how long, and what makes it wrong.

When this fires

A read path is slow and caching is being proposed; a cache already exists and data is stale or leaking across users; a review touches a TTL, a memoize decorator, a Redis client, or a CDN rule. It does not fire for a write path, or for a slow query nobody has profiled yet.

Procedure

1. Measure the uncached path first. Get a real number — p50 and p95, and how often it is called. Without it you cannot tell afterwards whether the cache helped, and you have no basis for the TTL. A cache added on a hunch is a guess with extra failure modes.
2. Try to make the cache unnecessary. A missing index, an N+1, a payload fetching columns nobody reads, or a call made in a loop are all cheaper to fix than to cache, and the fix has no staleness. Only cache what is genuinely expensive and genuinely re-read.
3. Classify the data by staleness tolerance, and let that pick the TTL — not the other way round. Ask the person who owns the data, in seconds: how long may this be wrong? Pricing, permissions, balances and anything a user just edited usually answer "not at all", which means either no cache or write-through invalidation, never a hopeful TTL.
4. Pick the layer deliberately. Each one fails differently — see below. Caching the same value at three layers means three different answers and three invalidation paths.
5. Design the key before the read. The key must contain every input that changes the value: identity or tenant, authorization scope, locale, feature flags, API version, and the query parameters that matter. A key missing the tenant is a cross-tenant data leak, and it will look exactly like a cache working well. Never key on a mutable object's memory identity.
6. Write the invalidation path in the same change as the read path. Decide which of these you are using and say so: TTL expiry only; explicit delete or overwrite on every write; or versioned keys (put a version or updated-at in the key so a change makes old entries unreachable and eviction cleans them up). Versioned keys are the least fragile — nothing to purge, no fan-out — and are the default worth reaching for.
7. Decide what happens when the cache is unavailable. A cache whose outage takes the app down is not a cache, it is an undeclared database. The read path should fall through to the source on error and timeout, with a short timeout, and the fallback must be load the source can actually survive.
8. Handle the stampede. When a hot key expires under load, every concurrent request misses at once and hits the source together. Mitigate with a single-flight lock per key, a jittered TTL so keys do not expire in lockstep, or serving stale while one worker refreshes.
9. Decide about negative results. Caching "not found" stops a hammering miss, but it also caches a race: a row created right after the negative entry stays invisible for the TTL. Cache negatives for seconds, not minutes, and invalidate them on create.
10. Make it observable. Emit hit rate, miss rate, and error/fallback count per cache. A hit rate you cannot see is a cache you cannot reason about, and a hit rate near zero means you have added a network hop and nothing else.
11. Re-measure, then prove invalidation. Write a value, read it back through the cached path, and confirm the new value appears. This is the step that is skipped and the one that catches real bugs — a fast wrong answer is worse than the slow right one.

The layers and how each one fails

• Browser / HTTP response cache. Cannot be invalidated, only expired — anything cached here is out of your hands until the TTL passes. Never cache a personalized or authorized response in a shared cache; get the public/private distinction wrong and one user's page is served to another. Version the URL when you need a hard cut.
• CDN / edge. Same exposure, plus: purges are eventually consistent across locations, and keying is by URL unless you declare which request headers vary the response. A missing vary declaration is the classic cross-user leak. Purging production edge cache is an outward-facing action — say what will be purged and ask before doing it.
• In-process (LRU, memoize, module-level dict). Cheapest and most local. But each instance holds its own truth, so N replicas give N answers and invalidation does not fan out; entries live in the heap and an unbounded one is a memory leak; it empties on deploy, which is why bugs here vanish when you restart and come back an hour later. Fine for immutable or derived-from-input data, dangerous for anything a user can edit.
• Shared cache (Redis, memcached). One truth for all instances and invalidation that fans out, at the cost of a network hop and a new dependency you must be able to lose. Under memory pressure it evicts — including keys you assumed were there — so nothing may be stored only in the cache. Flushing a shared production cache is destructive: it can stampede the database. Stop and ask before any flush; prefer deleting the specific keys.
• Database-side (materialized view, summary table). Consistent with the source at refresh time and queryable, but stale between refreshes and the refresh itself costs and may lock. Know what a refresh does to concurrent readers before scheduling one.
• Write-through denormalized column. Fastest read of all; the correctness burden moves onto every writer. One write path that forgets to update it — a migration, a backfill, an admin script, a second service — leaves the value permanently wrong with no TTL to heal it. Only take this on with a reconciliation job that can detect and repair drift.

Checklist

• Uncached cost measured, not assumed
• Cheaper non-cache fix considered and ruled out in one line
• Staleness tolerance stated in seconds by someone who owns the data
• Layer chosen, with its failure mode named
• Key contains tenant/identity, authorization scope, and every varying input
• Invalidation strategy written in the same change as the read
• Behaviour on cache outage and on stampede is defined
• Hit rate and fallback count are emitted
• Write-then-read-back actually executed and observed

Failure handling

• Stale data reported in production — do not raise the TTL question first. Find which layer is serving it: check the source, then each cache in front of it, and find where the answers diverge. The layer that diverges is the one with the broken invalidation.
• One user seeing another user's data — treat as a security incident, not a cache bug. Stop, report it, and check the key and any vary declaration before anything else.
• Hit rate near zero — the key varies more than the data does. Usually a timestamp, a request id, or a serialized object with unstable ordering is in the key. Remove the cache until the key is fixed; it is currently pure overhead.
• It got faster but you cannot say by how much — you skipped step 1. Say so rather than reporting an improvement you did not measure.
• Cannot reproduce staleness locally — expected, and not evidence it is fixed. Single-instance local runs hide per-instance cache divergence entirely. Say it is unreproduced locally.

Evidence to report

Before and after numbers from the same measurement, with the method named. The cache key format written out literally. The TTL and the staleness tolerance it came from. The invalidation path, and the result of the write-then-read-back check. Which layers now hold a copy of this value. What is still unverified — multi-instance behaviour, cache-down fallback, and stampede under load are the three most commonly claimed and least commonly tested.