Cache Sharding And Replication

Cache Sharding And Replication

Purpose

Decide how a cache is laid out across nodes, and what happens when one of those nodes goes away. This is a topology skill only: whether to cache, how long to keep an entry, and how to invalidate it are caching-strategies, and everything here assumes those decisions are already made.

The failure this prevents is the one that never looks like a cache incident. A cache node is restarted for a routine upgrade; with N balanced nodes, consistent hashing and no replicas, about 1/N of the keyspace loses its cached copy. After fail-fast remapping, requests for those keys can reach the origin until refill, while survivors continue serving. The cache tier reports a modest dip in hit rate. The database saturates. Nobody investigating the database is looking at the cache, because the cache is up.

Workflow

Before recommending changes, inspect cache product/version, Java client and resolved dependencies, runtime/toolchain, routing and retry configuration, replica placement, working-set bytes, per-node request share and origin capacity at the required SLO. This skill is language-independent and declares no Java baseline or executable Java examples; do not infer support for a client feature or authorize an upgrade. With missing measurements, provide conditional estimates and the exact measurement needed, not a production sizing or confirmed incident diagnosis.

1. Classify the cache first: performance or availability. If the origin cannot serve the full request rate with the cache empty, the cache is an availability component. Separate this from durability: the cache does not become the authoritative data store by being essential.
2. Do the node-loss arithmetic before choosing a topology. Measure the request share owned by each node; total_rate / N is only the uniform approximation. Losing node i can send its request share to the origin, plus secondary evictions and retries. Compare rate, concurrency, query mix and duration to the origin's measured capacity. The worked example is references/node-loss-and-origin-protection.md.
3. Choose sharding or full replication from the working set. If the whole working set fits comfortably in one node's memory and reads dominate, replicating everything can remove a network hop only when the replica is process-local. It avoids key loss after a node failure if routing and remaining capacity work, and costs roughly N × value memory plus metadata.
4. Choose the topology — client-sharded, proxy, or clustered — on operational cost and client complexity and measured end-to-end latency. The comparison is references/topologies.md.
5. Set the replication factor from step 2, not from a default. Replication exists here to keep the shard served when a node dies; if the arithmetic says the origin survives a node loss, RF=1 is a legitimate, cheaper answer. RF counts all copies including the primary; place them across the failure domains being protected and check promotion/quorum requirements.
6. Exercise failure under load—crash, partition/timeout, promotion and rejoin—and assert bounds on origin rate/concurrency, client errors and recovery, not only cache hit rate.
7. Add a local L1 only for a measured reason, and accept that invalidation now has to reach every instance's L1 as well as the shared tier.

Decision block

Sharded cache (each key has an owner shard, with optional replicas) when:
- the working set exceeds one node's memory, or memory cost makes N copies unattractive
- writes and invalidations are frequent enough that keeping N copies converged is work
Fully replicated cache (every node holds everything) when:
- the working set fits one node's memory with headroom, reads dominate heavily, and the
  value of surviving node loss exceeds N × memory; only process-local copies remove the hop
- typically the shape of small reference data: feature flags, rates, configuration
Replicate each shard (RF > 1) when:
- the measured node-loss arithmetic exceeds origin headroom
- or one shard is read-hot and the product can route reads to replicas within the required
  consistency model
Keep RF = 1 when:
- the origin demonstrably absorbs a node loss, and the memory is better spent on a larger
  working set; replicas can also serve reads if the product and consistency contract allow it
Prefer a proxy or a clustered cache over client-side sharding when:
- clients are polyglot, numerous, or cannot be redeployed together; the topology then
  changes without touching them
Prefer client-side sharding when:
- clients are few and share a runtime, and the extra network hop is a measurable share of
  the cache's own latency — the point of a cache is that it is fast
Do not add a cache node to fix a hot key:
- one key has one owner under every mapping function (hot-partitions-and-rebalancing)

Rules

• Losing a cache node is an origin-load event. Size the origin, or the protection in front of it, for the loss of one cache node — that is a routine occurrence (upgrade, eviction, spot reclaim), not a disaster scenario.
• A rolling restart can cause repeated remapping or replica promotion. Gate each next restart on origin headroom, client SLOs and restored replica readiness, not just recovered hit rate.
• Stable placement limits movement: hash(key) % N remaps nearly the entire keyspace on a membership change, turning one node's loss into a total miss storm. Consistent hashing is one option; fixed slots can also preserve placement. The mapping function belongs to consistent-hashing; this is the consequence.
• Node loss has a second-order cost when keys remap and refill on survivors: their memory did not grow, so insufficient headroom can raise evictions on previously healthy shards. Measure this effect rather than assuming the hit-rate dip equals the lost key share.
• Replication alone does not give read-your-writes. With asynchronous replication, a write acknowledged by one replica and a read served by another may return the old value. If the requirement is that a user sees their own change, route to a copy known to have applied that write, or bypass stale cache copies and read an origin endpoint that supplies the guarantee. Arbitrary replica affinity and invalidation alone are insufficient; promotion may lose an acknowledged write. consistency-models owns the guarantee, including the behavior across failover.
• Replication guarantees are product/configuration-specific. Asynchronous replicas have no useful convergence deadline unless lag is bounded and monitored. TTL bounds how long a missed invalidation can survive only if expiry forces a correct reload; it is not a consistency proof.
• Client-side sharding puts the topology in every client. Adding a node means every client must use compatible node lists, virtual-node counts and hashes; overlapping versions need an explicit migration/invalidation protocol. A disagreement is two clients writing the same key to two different nodes, and both of them may read stale. Distribute membership through one source, versioned; versioning alone does not make an overlapping rollout coherent.
• A proxy costs one extra network hop on the cache path, which is the path chosen for being fast. Measure the hop against T_source before rejecting it: a fraction of a millisecond in front of a source costing tens of milliseconds is usually the right trade, and it buys topology changes without client deploys.
• A clustered cache with server-owned placement moves membership out of application config. Cross-slot multi-key semantics vary by product; Redis Cluster rejects many such operations, while other systems coordinate them at extra latency/availability cost. Check the exact command and failure contract against the access pattern.
• A near-cache (local L1 in front of the shared L2) is a second cache with its own coherence problem, and it is per-instance: invalidating the L2 invalidates no L1. caching-strategies owns invalidation propagation and the L1 TTL as the safety net; the topology consequence is that the copies to invalidate now number instances + replicas.
• Every entry crossing the network is serialised, so the value size is a throughput decision, not a detail. A large value multiplied by the fan-out of a warm-up is a network incident — serialization-performance owns the format cost.

Deliverable

Return the chosen topology and rejected alternative, measured inputs versus assumptions, node-loss origin-load estimate, replica placement/read policy, and failure-test acceptance bounds with rollout abort criteria. For incidents, distinguish observed timing/counters from the cache-loss hypothesis and name the load test or evidence that would refute it. State which checks actually ran; a paper estimate is not demonstrated failure tolerance.

Primary sources

• Redis Cluster specification
• Redis replication
• Amazon Dynamo paper

References

• Cache topologies — client-side sharded, proxy-fronted, clustered and fully replicated compared on failure behaviour, operational cost, client complexity and consistency, with the near-cache layer and a decision table. Read when choosing or changing a topology, or when a client library's sharding is in question.
• Node loss and origin protection — the miss storm computed from real numbers, replication factor as the lever, warming, coalescing, origin admission control, and the kill-a-node-under-load test with the bound it asserts. Read before sizing a cache tier, after any incident where the origin saturated, or when planning a cache upgrade or restart.