Redis

Redis — cache, lock, rate limiter, queue done correctly

Redis is four primitives wearing one server: a cache, a distributed lock, a rate limiter, and a job queue. Each has a correctness contract that has nothing to do with whether SET returns OK. You almost certainly already "have Redis working" — a value goes in, a value comes out. What you do not yet have is a cache that won't stampede your database, a lock that won't release someone else's hold, a rate limiter that survives concurrent requests, and a queue whose jobs aren't silently lost when a worker dies. This skill is engine- and pattern-level, client-SDK-agnostic: examples are redis-cli and Lua, so they hold across node-redis, ioredis, redis-py, go-redis, and Lettuce. Every pattern applies unchanged to Valkey and any Redis-protocol store, since Valkey is a BSD fork of Redis 7.2 and stays protocol-compatible.

When to use / When NOT

Use when you are:

• Adding or reviewing a cache: TTL strategy, cache-aside vs write-through, key naming, stampede/thundering-herd prevention, negative caching, invalidation.
• Building a distributed lock: "only one worker does X", cron de-duplication, leader-ish mutual exclusion — the SET NX PX + Lua-release + fencing-token contract, and the Redlock call.
• Building a rate limiter: per-user/IP/API-key throttling (fixed window, sliding-window log, sliding-window counter, token bucket) and why naive INCR+EXPIRE is a race.
• Running a queue / background jobs on Redis: Lists vs Streams vs a library (BullMQ, Sidekiq, RQ, Celery), at-least-once delivery, acks, stalled-job recovery, idempotency.
• Deciding eviction & persistence: maxmemory-policy, RDB vs AOF, cache mode vs database mode.
• Reviewing Redis code for foot-guns: KEYS *, unbounded keys, non-atomic read-modify-write, missing TTL, a lock released by the wrong owner.

NOT for (route instead):

• Durable SQL queue (SELECT … FOR UPDATE SKIP LOCKED), engine-level SQL caching → ../postgresdb/SKILL.md. Boundary: if the durable store is the queue, that's postgresdb; if Redis is the queue, you're here.
• Vector similarity / semantic search as the primary job → ../vector-db/SKILL.md. Redis 8 vector sets are only noted here.
• Analytical / clickstream event store → ../clickhouse-analytics/SKILL.md.
• System-level capacity / load strategy, and app/CDN/HTTP-layer caching (Cache-Control, ISR, edge) → ../scaling/SKILL.md. This skill owns the Redis-specific data contract underneath.
• Provisioning a managed Redis (cluster sizing, dashboards) → ../fly-io/SKILL.md / ../aws-essentials/SKILL.md. This skill gives the client-side contract, not console clicks.
• Language-runtime async/job concepts unrelated to Redis → ../nodejs/SKILL.md or your language skill.

Non-negotiables

1. Every cache key has a TTL. No TTL in a cache = a memory leak that eventually evicts or 500s.
2. Never KEYS * in production — use SCAN. KEYS is O(N) and blocks the single thread; one bad pattern freezes every client.
3. Read-modify-write must be atomic. App-side GET then SET/INCR races under concurrency. Use one Lua EVAL, a single atomic command, or WATCH/MULTI.
4. A lock = random token + PX + Lua compare-and-delete. Plain DEL releases whoever holds it now (maybe not you); a lock with no PX is held forever if the owner crashes.
5. Pick maxmemory-policy deliberately. The default noeviction makes a full cache fail every write — correct for a database, fatal for a cache.
6. A queue must ack and recover. RPOP straight into a worker loses the job if the worker dies mid-task; use a processing list or Streams consumer groups.
7. Cap unbounded structures. Lists grow forever without LTRIM; streams without MAXLEN ~.
8. One Lua = one short atomic step. Redis is single-threaded; a long script or huge MULTI/EXEC blocks all clients, not just yours.
9. Namespace keys as svc:entity:id (e.g. cart:user:42) so scans, eviction, and humans can reason.
10. Know cache vs database durability before choosing persistence. Cache = lose-it-and-rebuild; database = RDB/AOF and noeviction. Decide which Redis is before you tune it.

Decision table — which primitive, which structure

Caching

Namespace + TTL with jitter. A fixed TTL on a batch of keys written together synchronizes their expiry — they all die in the same second and stampede the origin at once. Add jitter.

# Bad: 1000 keys warmed in a loop with the same TTL all expire together
SET product:42 "$json" EX 300
# Good: spread expiry with per-key jitter (300s ± up to 60s)
SET product:42 "$json" EX $(( 300 + RANDOM % 60 ))

Cache-aside (read path). Why: it is the default for read-heavy data; the trap is forgetting the TTL and forgetting to cache the miss.

# Bad: cache only hits, never the miss → every request for an absent id hits the DB
val = GET product:42
if val == nil: val = db.fetch(42); SET product:42 val EX 300   # absent rows still hammer the DB

# Good: cache-aside with negative caching
val = GET product:42
if val == "__MISS__": return null                              # cached miss, no DB call
if val == nil:
    val = db.fetch(42)
    if val == null: SET product:42 "__MISS__" EX 30            # short negative TTL
    else:           SET product:42 val EX 300

Stampede prevention. When a hot key expires, N concurrent requests all miss and hammer the origin. Two mitigations:

• Recompute lock — one request rebuilds, the rest briefly serve stale or wait. The lock TTL must exceed worst-case origin latency, or two requests rebuild anyway.

  # On miss, try to claim the rebuild; only the winner recomputes
  SET product:42:lock 1 NX PX 5000   # 5s > slowest DB rebuild
  # winner: rebuild, SET product:42, then DEL product:42:lock
  # losers: serve last-known value or retry after a short sleep
  

• Probabilistic early expiration (XFetch) — refresh before TTL with rising probability so one request renews early while others still hit a warm key. Math + write-through/read-through/ write-behind in references/caching.md.

Invalidation: on write, either delete the key (next read rebuilds) or write-through. Never rely on manual invalidation instead of a TTL — you will miss a path and serve stale forever.

Distributed locks

# Bad: GET-then-DEL races — between your GET and DEL the lock can expire and a NEW owner takes it,
#      and your DEL frees THEIR lock.
GET resource:lock        # == my token?
DEL resource:lock        # ← may delete someone else's lock

# Good: claim with a random token + PX, release only if the token is still mine (atomic Lua).
SET resource:lock "$TOKEN" NX PX 30000     # NX = only if absent; PX = auto-expire so a crash frees it

Release must be a compare-and-delete in one Lua step:

-- KEYS[1] = lock key, ARGV[1] = my token. Returns 1 if I held and released it, else 0.
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end

TTL vs work duration. If the protected work can outlast PX, the lock expires mid-task and a second worker starts. Either size PX above the worst case, or run a watchdog that renews the TTL (again via a token-checked Lua PEXPIRE) while work continues — never blindly PEXPIRE.

Fencing tokens. A GC pause or a long syscall can freeze the lock holder past expiry; another worker acquires, then the paused one wakes and writes — both think they hold the lock. The only fix is a fencing token: a monotonically increasing number issued with the lock and checked by the protected resource, which rejects any write carrying a stale token. No Redis lock alone provides this.

Redlock decision. Single-instance SET NX PX is fine when occasional double-execution is tolerable (idempotent work, best-effort cron de-dup). For hard mutual exclusion, a single Redis lock is not enough and multi-instance Redlock is contested (it assumes bounded clock drift and no long pauses) — fence the resource or use a lease/consensus system. Full Redlock walk-through and critique in references/locks-and-rate-limiting.md.

Rate limiting

# Bad: GET/INCR then EXPIRE as two steps. If the process dies (or loses the race) between INCR and
#      EXPIRE, the key has NO TTL and counts forever — the user is throttled permanently.
n = INCR rl:user:42
if n == 1: EXPIRE rl:user:42 60     # ← may never run

Fixed window, done atomically in one EVAL:

-- KEYS[1] = counter, ARGV[1] = limit, ARGV[2] = window seconds. Returns 1 = allow, 0 = deny.
local n = redis.call('INCR', KEYS[1])
if n == 1 then redis.call('EXPIRE', KEYS[1], ARGV[2]) end   -- TTL set atomically with the first hit
if n > tonumber(ARGV[1]) then return 0 end
return 1

Fixed window allows up to 2× the limit across a boundary. For smoother limits:

• Sliding-window log — a sorted set scored by timestamp, pruned and counted in one Lua: ZREMRANGEBYSCORE (drop entries older than the window) → ZCARD (count) → ZADD (record now) → EXPIRE. Exact, but O(requests) memory per key.
• Token bucket — a hash holding {tokens, last_refill}; refill by elapsed time and decrement, all in one Lua. Allows bursts up to bucket size, then a steady rate. Smallest memory.

Choose by accuracy-vs-memory; full Lua for all four in references/locks-and-rate-limiting.md.

Queues

Reliable list: never RPOP straight into the worker — a crash mid-task loses the job. Move it to a per-worker processing list atomically, then ack by removing it.

job=$(redis-cli LMOVE jobs jobs:proc:w1 LEFT RIGHT)   # atomic: pop from jobs, push to processing
# ... do work (idempotently) ...
redis-cli LREM jobs:proc:w1 1 "$job"                  # ack: remove from processing
# a reaper requeues anything left in jobs:proc:* after a worker dies

Streams (preferred for reliable jobs): consumer groups give per-message acks and a Pending Entries List for recovery.

redis-cli XGROUP CREATE jobs g1 '$' MKSTREAM
redis-cli XREADGROUP GROUP g1 worker1 COUNT 10 BLOCK 5000 STREAMS jobs '>'
# ... process ...
redis-cli XACK jobs g1 "$id"                          # ack a done message
redis-cli XAUTOCLAIM jobs g1 worker2 60000 0          # reclaim messages idle > 60s (stalled worker)

Delivery is at-least-once: a message can be reprocessed after a crash, so make handlers idempotent (dedupe on a job id / idempotency key). Reach for BullMQ (TypeScript; stalled-job lock renewal via lockDuration/lockRenewTime), Sidekiq (Ruby), or RQ/Celery (Python) when you want retries, delays, and a dashboard — don't reinvent them. Full implementations, DLQ, and the library comparison in references/queues.md.

Eviction & persistence

A "cache" running noeviction with no TTLs is a memory leak that starts 500ing writes when full — the single most common Redis incident. Set maxmemory and an allkeys-* policy for a cache.

Persistence: RDB = periodic point-in-time snapshots (fast restart, can lose the last interval); AOF = append every write (durable, slower, larger). A pure cache often needs neither — losing it just rebuilds from the origin. Decide cache-mode vs database-mode first, then persistence follows.

Anti-patterns → STOP

Quick reference

# Cache:      SET k v EX 300 | GETEX k EX 300 | DEL k | SCAN 0 MATCH 'product:*' COUNT 100
# Lock:       SET k tok NX PX 30000  (release via the Lua compare-and-delete above)
# Rate limit: EVAL "<lua>" 1 rl:user:42 100 60   (limit 100 / 60s)
# Queue:      LMOVE jobs jobs:proc LEFT RIGHT | LREM jobs:proc 1 "$job"
#             XADD jobs '*' field v | XREADGROUP ... | XACK | XAUTOCLAIM

# Diagnostics (read-only):
redis-cli INFO memory          # used_memory, maxmemory, evicted_keys
redis-cli --bigkeys            # find the keys eating your memory
redis-cli SLOWLOG GET 10       # slowest recent commands
redis-cli OBJECT FREQ mykey    # access frequency (needs allkeys-lfu)
redis-cli MEMORY USAGE mykey   # bytes for one key
redis-cli SCAN 0 MATCH 'sess:*' COUNT 100   # never KEYS in prod

scripts/verify.sh scans your project for the high-confidence foot-guns (KEYS * in source, lock release without a token compare, INCR with no EXPIRE, maxmemory with no policy). It is read-only, never connects to a server, and exits 0 when no Redis usage is found.

Project grounding

Record this project's Redis decisions in 02-DOCS/wiki/stack/redis.md (recorded, not gated — same convention postgresdb uses): which primitive(s) you run, the maxmemory-policy, persistence (RDB/AOF/none), the client library, and any lock/rate-limit Lua you depend on. Future agents read this before touching the cache.

See Also

• references/caching.md — read/write-through, write-behind, XFetch math, negative caching, hot/big keys.
• references/locks-and-rate-limiting.md — full Redlock + critique + fencing, watchdog renewal, all four rate-limit algorithms with Lua.
• references/queues.md — full reliable-list + Streams implementations, XAUTOCLAIM recovery, DLQ, BullMQ/Sidekiq/RQ/Celery detail, idempotency.
• Siblings: ../postgresdb/SKILL.md · ../vector-db/SKILL.md · ../scaling/SKILL.md · ../clickhouse-analytics/SKILL.md.