Caching

Caching Rules

1. Cache Strategy Selection

When to Cache

Cache Placement

Two-Level Cache Pattern

Request → L1 (local) → L2 (distributed) → Data source
             ↑ miss         ↑ miss              │
             └──────────────┴───── populate ─────┘

• L1 (Caffeine, Guava): sub-millisecond, per-instance
• L2 (Redis, Memcached): single-digit ms, shared
• On L1 miss, check L2 before hitting the data source
• Invalidate both levels on writes

2. TTL Design Guidelines

Recommended TTL by Data Type

TTL Rules

• Always set a TTL — never cache indefinitely
• Shorter TTL for data that affects user experience when stale
• Longer TTL for reference data that changes infrequently
• Use explicit eviction on writes in addition to TTL
• Consider TTL jitter to avoid cache stampede

TTL Jitter

Effective TTL = base_ttl + random(0, jitter_range)

Example: base 10min, jitter 2min → actual TTL between 10-12min

Without jitter, cached entries expire simultaneously, causing a thundering herd to the data source.

3. Cache Key Design

Key Structure

<namespace>:<entity>:<identifier>[:<variant>]

Examples:
  user:profile:12345
  product:detail:SKU-001:v2
  search:results:shoes:page=1:size=20

Key Design Rules

4. Cache Invalidation Patterns

Write-Through

Update the cache entry simultaneously when writing to the data store. Ensures cache is always consistent but adds write latency.

Write request → Update DB → Update cache → Return

Evict on Write (Cache-Aside)

Evict the cache entry on write and let the next read repopulate it. Simpler to implement and avoids stale data from failed cache updates.

Write: Write request → Update DB → Evict cache → Return
Read:  Read request → Cache miss → Read DB → Populate cache → Return

Write-Behind (Write-Back)

Write to cache first, then asynchronously persist to the data store. Lowest write latency but risks data loss.

Write request → Update cache → Return (async persist to DB)

Event-Based Invalidation

Invalidate cache entries via events (message queue, application events) for cross-service consistency. Best for distributed systems where multiple services share cached data.

Invalidation Rules

• Prefer explicit eviction over TTL-only for mutable data
• Invalidate on all write paths (update, delete, bulk operations)
• Use event-based invalidation for cross-service cache consistency
• Test cache invalidation paths — stale cache bugs are hard to diagnose

5. Cache Stampede Prevention

A cache stampede occurs when many requests simultaneously miss the cache and hit the data source.

Prevention Strategies

Probabilistic Early Recomputation

Instead of waiting for exact expiry, each request probabilistically decides whether to recompute. As TTL approaches, the probability of recomputation increases.

should_recompute = (current_time - (expiry - ttl * beta * log(rand()))) > 0

beta: tuning factor (typically 1.0)
rand(): uniform random in (0, 1]

• When beta = 1.0, roughly 1 request recomputes before expiry under normal load
• Higher beta increases early recomputation probability
• No coordination overhead — purely probabilistic, no locks required
• Best suited for high-throughput keys where occasional early recomputation is acceptable

Locking Strategy Details

1. Thread A: cache miss → acquire lock → recompute → populate cache → release lock
2. Thread B: cache miss → lock unavailable → wait or return stale value

• Blocking lock: waiters block until the lock holder populates the cache; simple but increases tail latency
• Try-lock with stale fallback: if lock is unavailable, serve the stale value; requires keeping the expired entry accessible
• Distributed lock (e.g., Redis SET NX EX): required when multiple instances share the cache; set a short lock TTL to avoid deadlocks

Request Coalescing (Singleflight)

Multiple concurrent requests for the same key are collapsed into a single data-source call. Only the first request triggers computation; subsequent requests wait for and share the result.

Request A ─┐
Request B ──┼─→ single fetch → result shared to all
Request C ─┘

• Eliminates redundant calls entirely, not probabilistically
• Must be implemented in the application layer (e.g., Go singleflight, custom deduplication map)
• Scope is per-instance; combine with distributed locking for multi-instance protection

6. Cache Warming

When to Warm

• Application startup with predictable hot data
• After cache flush or failover
• Before traffic spike (scheduled events, promotions)

Warming Rules

• Warm only high-frequency keys — not the entire dataset
• Stagger warming to avoid overwhelming the data source
• Set shorter TTL for warmed entries if staleness risk is high

7. Anti-Patterns

• Caching without TTL (memory leak risk)
• Caching mutable objects (caller modifies cached reference)
• Cache key collisions (using only entity ID without type prefix)
• Caching null values without explicit handling
• Ignoring cache in delete/update paths (stale reads)
• Over-caching (caching everything "just in case")
• No monitoring of cache hit/miss rates
• Using distributed cache for data that only needs local caching
• Cache-aside without stampede protection on hot keys
• Storing large objects (> 1MB) in distributed cache without compression

8. Cache Patterns Comparison

Sections 1 and 4 introduce individual patterns. This section provides a systematic comparison to guide pattern selection.

Read Patterns

Write Patterns

Pattern Selection Guide

Read-heavy, simple setup?           → Cache-Aside
Read-heavy, want transparent cache? → Read-Through
Need write consistency?             → Write-Through
Need fast writes, tolerate risk?    → Write-Behind
High read + high write?             → Write-Behind + Read-Through

Combining Read and Write Patterns

• Read-Through + Write-Through: full cache transparency with strong consistency; higher write latency
• Read-Through + Write-Behind: full cache transparency with fast writes; eventual consistency for writes
• Cache-Aside + Write-Through: app controls reads, cache stays consistent on writes; most common combination

9. Distributed Cache Patterns

Consistent Hashing

Maps cache keys to nodes on a hash ring. When a node is added or removed, only a fraction of keys are remapped.

Hash Ring:

    Node A (0°)
       ╱          ╲
  Node D (270°)    Node B (90°)
       ╲          ╱
    Node C (180°)

Key "user:123" hashes to 110° → assigned to Node C (next node clockwise)
Node B removed → only keys between A and B remap to C

• Virtual nodes: each physical node maps to multiple points on the ring, improving key distribution
• Without virtual nodes, adding/removing nodes causes uneven load
• Commonly used by Redis Cluster, Memcached client libraries, and application-level sharding

Near Cache (L1/L2 Architecture)

Extends the two-level cache concept (Section 1) with consistency mechanisms for distributed environments.

Consistency Strategies for Near Cache

• Invalidation events: L2 publishes invalidation messages (e.g., Redis Pub/Sub); L1 evicts on receive
• Short L1 TTL: keep L1 TTL short (seconds) to limit staleness window without events
• Version check: L1 stores version tag; on read, compare with L2 version before serving

Cache Replication vs Partitioning

• Hybrid approach: partition data across shards, replicate each shard (e.g., Redis Cluster with replicas)
• Replication lag causes temporary inconsistency; design reads to tolerate this

10. Cache Sizing and Monitoring

Cache Sizing Guidelines

Sizing Formula

Required memory ≈ working_set_count × avg_serialized_size × (1 + overhead_factor)

overhead_factor: ~0.1–0.3 depending on data structure and cache implementation

• Start with a conservative estimate, then adjust based on actual hit rate and eviction metrics
• For distributed caches, divide by partition count to get per-node sizing

Key Metrics to Monitor

Monitoring Rules

• Track hit rate per cache region or namespace, not just globally
• Set up alerts for hit rate drops — they often indicate a bug, not a capacity issue
• Monitor eviction rate alongside memory usage; high eviction with low memory means the cache is undersized
• Track latency percentiles (p50, p95, p99), not just averages
• Log cache misses for hot keys to identify stampede candidates

Additional References

• For Redis caching strategies, cache-aside, write-through, TTL management, and eviction policies, see references/redis-patterns.md
• For Spring Boot implementation patterns (Caffeine, Redis, @Cacheable), see spring-framework skill — references/caching.md