Redis State Management
A comprehensive skill for mastering Redis state management patterns in distributed systems. This skill covers caching strategies, session management, pub/sub messaging, distributed locks, data structures, and production-ready patterns using redis-py.
When to Use This Skill
Use this skill when:
- Implementing high-performance caching layers for web applications
- Managing user sessions in distributed environments
- Building real-time messaging and event distribution systems
- Coordinating distributed processes with locks and synchronization
- Storing and querying structured data with Redis data structures
- Optimizing application performance with Redis
- Scaling applications horizontally with shared state
- Implementing rate limiting, counters, and analytics
- Building microservices with Redis as a communication layer
- Managing temporary data with automatic expiration (TTL)
- Implementing leaderboards, queues, and real-time features
Core Concepts
Redis Fundamentals
Redis (Remote Dictionary Server) is an in-memory data structure store used as:
- Database: Persistent key-value storage
- Cache: High-speed data layer
- Message Broker: Pub/sub and stream messaging
- Session Store: Distributed session management
Key Characteristics:
- In-memory storage (microsecond latency)
- Optional persistence (RDB snapshots, AOF logs)
- Rich data structures beyond key-value
- Atomic operations on complex data types
- Built-in replication and clustering
- Pub/sub messaging support
- Lua scripting for complex operations
- Pipelining for batch operations
Redis Data Structures
Redis provides multiple data types for different use cases:
Strings: Simple key-value pairs, binary safe
- Use for: Cache values, counters, flags, JSON objects
- Max size: 512 MB
- Commands: SET, GET, INCR, APPEND
Hashes: Field-value maps (objects)
- Use for: User profiles, configuration objects, small entities
- Efficient for storing objects with multiple fields
- Commands: HSET, HGET, HMGET, HINCRBY
Lists: Ordered collections (linked lists)
- Use for: Queues, activity feeds, recent items
- Operations at head/tail are O(1)
- Commands: LPUSH, RPUSH, LPOP, RPOP, LRANGE
Sets: Unordered unique collections
- Use for: Tags, unique visitors, relationships
- Set operations: union, intersection, difference
- Commands: SADD, SMEMBERS, SISMEMBER, SINTER
Sorted Sets: Ordered sets with scores
- Use for: Leaderboards, time-series, priority queues
- Range queries by score or rank
- Commands: ZADD, ZRANGE, ZRANGEBYSCORE, ZRANK
Streams: Append-only logs with consumer groups
- Use for: Event sourcing, activity logs, message queues
- Built-in consumer group support
- Commands: XADD, XREAD, XREADGROUP
Connection Management
Connection Pools:
Redis connections are expensive to create. Always use connection pools:
import redis
# Connection pool (recommended)
pool = redis.ConnectionPool(host='localhost', port=6379, db=0, max_connections=10)
r = redis.Redis(connection_pool=pool)
# Direct connection (avoid in production)
r = redis.Redis(host='localhost', port=6379, db=0)
Best Practices:
- Use connection pools for all applications
- Set appropriate max_connections based on workload
- Enable decode_responses=True for string data
- Configure socket_timeout and socket_keepalive
- Handle connection errors with retries
Data Persistence
Redis offers two persistence mechanisms:
RDB (Redis Database): Point-in-time snapshots
- Compact binary format
- Fast restart times
- Lower disk I/O
- Potential data loss between snapshots
AOF (Append-Only File): Log of write operations
- Better durability (fsync policies)
- Larger files, slower restarts
- Can be automatically rewritten/compacted
- Minimal data loss potential
Hybrid Approach: RDB + AOF for best of both worlds
RESP 3 Protocol
Redis Serialization Protocol version 3 offers:
- Client-side caching support
- Better data type support
- Push notifications
- Performance improvements
import redis
from redis.cache import CacheConfig
# Enable RESP3 with client-side caching
r = redis.Redis(host='localhost', port=6379, protocol=3,
cache_config=CacheConfig())
Caching Strategies
Cache-Aside (Lazy Loading)
Pattern: Application checks cache first, loads from database on miss
import redis
import json
from typing import Optional, Dict, Any
r = redis.Redis(decode_responses=True)
def get_user(user_id: int) -> Optional[Dict[str, Any]]:
"""Cache-aside pattern for user data."""
cache_key = f"user:{user_id}"
# Try cache first
cached_data = r.get(cache_key)
if cached_data:
return json.loads(cached_data)
# Cache miss - load from database
user_data = database.get_user(user_id) # Your DB query
if user_data:
# Store in cache with 1 hour TTL
r.setex(cache_key, 3600, json.dumps(user_data))
return user_data
Advantages:
- Only requested data is cached (efficient memory usage)
- Cache failures don't break the application
- Simple to implement
Disadvantages:
- Cache miss penalty (latency spike)
- Thundering herd on popular items
- Stale data until cache expiration
Write-Through Cache
Pattern: Write to cache and database simultaneously
def update_user(user_id: int, user_data: Dict[str, Any]) -> bool:
"""Write-through pattern for user updates."""
cache_key = f"user:{user_id}"
# Write to database first
success = database.update_user(user_id, user_data)
if success:
# Update cache immediately
r.setex(cache_key, 3600, json.dumps(user_data))
return success
Advantages:
- Cache always consistent with database
- No read penalty for recently written data
Disadvantages:
- Write latency increases
- Unused data may be cached
- Extra cache write overhead
Write-Behind (Write-Back) Cache
Pattern: Write to cache immediately, sync to database asynchronously
import redis
import json
from queue import Queue
from threading import Thread
r = redis.Redis(decode_responses=True)
write_queue = Queue()
def async_writer():
"""Background worker to sync cache to database."""
while True:
user_id, user_data = write_queue.get()
try:
database.update_user(user_id, user_data)
except Exception as e:
# Log error, potentially retry
print(f"Failed to write user {user_id}: {e}")
finally:
write_queue.task_done()
# Start background writer
Thread(target=async_writer, daemon=True).start()
def update_user_fast(user_id: int, user_data: Dict[str, Any]):
"""Write-behind pattern for fast writes."""
cache_key = f"user:{user_id}"
# Write to cache immediately (fast)
r.setex(cache_key, 3600, json.dumps(user_data))
# Queue database write (async)
write_queue.put((user_id, user_data))
Advantages:
- Minimal write latency
- Can batch database writes
- Handles write spikes
Disadvantages:
- Risk of data loss if cache fails
- Complex error handling
- Consistency challenges
Cache Invalidation Strategies
Time-based Expiration (TTL):
# Set key with expiration
r.setex("session:abc123", 1800, session_data) # 30 minutes
# Or set TTL on existing key
r.expire("user:profile:123", 3600) # 1 hour
# Check remaining TTL
ttl = r.ttl("user:profile:123")
Event-based Invalidation:
def update_product(product_id: int, product_data: dict):
"""Invalidate cache on update."""
# Update database
database.update_product(product_id, product_data)
# Invalidate related caches
r.delete(f"product:{product_id}")
r.delete(f"product_list:category:{product_data['category']}")
r.delete("products:featured")
Pattern-based Invalidation:
# Delete all keys matching pattern
def invalidate_user_cache(user_id: int):
"""Invalidate all cache entries for a user."""
pattern = f"user:{user_id}:*"
# Find and delete matching keys
for key in r.scan_iter(match=pattern, count=100):
r.delete(key)
Cache Stampede Prevention
Problem: Multiple requests simultaneously miss cache and query database
Solution 1: Probabilistic Early Expiration
import time
import random
def get_with_early_expiration(key: str, ttl: int = 3600, beta: float = 1.0):
"""Prevent stampede with probabilistic early recomputation."""
value = r.get(key)
if value is None:
# Cache miss - compute and cache
value = compute_value(key)
r.setex(key, ttl, value)
return value
# Check if we should recompute early
current_time = time.time()
delta = current_time - float(r.get(f"{key}:timestamp") or 0)
expiry = ttl * random.random() * beta
if delta > expiry:
# Recompute in background
value = compute_value(key)
r.setex(key, ttl, value)
r.set(f"{key}:timestamp", current_time)
return value
Solution 2: Locking
from contextlib import contextmanager
@contextmanager
def cache_lock(key: str, timeout: int = 10):
"""Acquire lock for cache computation."""
lock_key = f"{key}:lock"
identifier = str(time.time())
# Try to acquire lock
if r.set(lock_key, identifier, nx=True, ex=timeout):
try:
yield True
finally:
# Release lock
if r.get(lock_key) == identifier:
r.delete(lock_key)
else:
yield False
def get_with_lock(key: str):
"""Use lock to prevent stampede."""
value = r.get(key)
if value is None:
with cache_lock(key) as acquired:
if acquired:
# We got the lock - compute value
value = compute_value(key)
r.setex(key, 3600, value)
else:
# Someone else is computing - wait and retry
time.sleep(0.1)
value = r.get(key) or compute_value(key)
return value
Session Management
Distributed Session Storage
Basic Session Management:
import redis
import json
import uuid
from datetime import datetime, timedelta
r = redis.Redis(decode_responses=True)
class SessionManager:
def __init__(self, ttl: int = 1800):
"""Session manager with Redis backend.
Args:
ttl: Session timeout in seconds (default 30 minutes)
"""
self.ttl = ttl
def create_session(self, user_id: int, data: dict = None) -> str:
"""Create new session and return session ID."""
session_id = str(uuid.uuid4())
session_key = f"session:{session_id}"
session_data = {
"user_id": user_id,
"created_at": datetime.utcnow().isoformat(),
"data": data or {}
}
r.setex(session_key, self.ttl, json.dumps(session_data))
return session_id
def get_session(self, session_id: str) -> dict:
"""Retrieve session data and refresh TTL."""
session_key = f"session:{session_id}"
session_data = r.get(session_key)
if session_data:
# Refresh TTL on access (sliding expiration)
r.expire(session_key, self.ttl)
return json.loads(session_data)
return None
def update_session(self, session_id: str, data: dict) -> bool:
"""Update session data."""
session_key = f"session:{session_id}"
session_data = self.get_session(session_id)
if session_data:
session_data["data"].update(data)
r.setex(session_key, self.ttl, json.dumps(session_data))
return True
return False
def delete_session(self, session_id: str) -> bool:
"""Delete session (logout)."""
session_key = f"session:{session_id}"
return r.delete(session_key) > 0
Session with Hash Storage
More efficient for session objects:
class HashSessionManager:
"""Session manager using Redis hashes for better performance."""
def __init__(self, ttl: int = 1800):
self.ttl = ttl
def create_session(self, user_id: int, **kwargs) -> str:
"""Create session using hash."""
session_id = str(uuid.uuid4())
session_key = f"session:{session_id}"
# Store as hash for efficient field access
session_fields = {
"user_id": str(user_id),
"created_at": datetime.utcnow().isoformat(),
**{k: str(v) for k, v in kwargs.items()}
}
r.hset(session_key, mapping=session_fields)
r.expire(session_key, self.ttl)
return session_id
def get_field(self, session_id: str, field: str) -> str:
"""Get single session field efficiently."""
session_key = f"session:{session_id}"
value = r.hget(session_key, field)
if value:
r.expire(session_key, self.ttl) # Refresh TTL
return value
def set_field(self, session_id: str, field: str, value: str) -> bool:
"""Update single session field."""
session_key = f"session:{session_id}"
if r.exists(session_key):
r.hset(session_key, field, value)
r.expire(session_key, self.ttl)
return True
return False
def get_all(self, session_id: str) -> dict:
"""Get all session fields."""
session_key = f"session:{session_id}"
data = r.hgetall(session_key)
if data:
r.expire(session_key, self.ttl)
return data
User Activity Tracking
def track_user_activity(user_id: int, action: str):
"""Track user activity with automatic expiration."""
activity_key = f"user:{user_id}:activity"
timestamp = datetime.utcnow().isoformat()
# Add activity to list
r.lpush(activity_key, json.dumps({"action": action, "timestamp": timestamp}))
# Keep only last 100 activities
r.ltrim(activity_key, 0, 99)
# Set expiration (30 days)
r.expire(activity_key, 2592000)
def get_recent_activity(user_id: int, limit: int = 10) -> list:
"""Get recent user activities."""
activity_key = f"user:{user_id}:activity"
activities = r.lrange(activity_key, 0, limit - 1)
return [json.loads(a) for a in activities]
Pub/Sub Patterns
Basic Publisher/Subscriber
Publisher:
import redis
r = redis.Redis(decode_responses=True)
def publish_event(channel: str, message: dict):
"""Publish event to channel."""
import json
r.publish(channel, json.dumps(message))
# Example usage
publish_event("notifications", {
"type": "user_signup",
"user_id": 12345,
"timestamp": datetime.utcnow().isoformat()
})
Subscriber:
import redis
import json
def handle_message(message):
"""Process received message."""
data = json.loads(message['data'])
print(f"Received: {data}")
# Initialize pubsub
r = redis.Redis(decode_responses=True)
p = r.pubsub()
# Subscribe to channels
p.subscribe('notifications', 'alerts')
# Listen for messages
for message in p.listen():
if message['type'] == 'message':
handle_message(message)
Pattern-Based Subscriptions
# Subscribe to multiple channels with patterns
p = r.pubsub()
p.psubscribe('user:*', 'notification:*')
# Get messages from pattern subscriptions
for message in p.listen():
if message['type'] == 'pmessage':
channel = message['channel']
pattern = message['pattern']
data = message['data']
print(f"Pattern {pattern} matched {channel}: {data}")
Async Pub/Sub with Background Thread
import redis
import time
r = redis.Redis(decode_responses=True)
p = r.pubsub()
def message_handler(message):
"""Handle messages in background thread."""
print(f"Handler received: {message['data']}")
# Subscribe with handler
p.subscribe(**{'notifications': message_handler, 'alerts': message_handler})
# Run in background thread
thread = p.run_in_thread(sleep_time=0.001)
# Publish some messages
r.publish('notifications', 'Hello!')
r.publish('alerts', 'Warning!')
time.sleep(1)
# Stop background thread
thread.stop()
Async Pub/Sub with asyncio
import asyncio
import redis.asyncio as redis
async def reader(channel: redis.client.PubSub):
"""Async message reader."""
while True:
message = await channel.get_message(ignore_subscribe_messages=True, timeout=None)
if message is not None:
print(f"Received: {message}")
# Stop on specific message
if message["data"].decode() == "STOP":
break
async def pubsub_example():
"""Async pub/sub example."""
r = await redis.from_url("redis://localhost")
async with r.pubsub() as pubsub:
# Subscribe to channels
await pubsub.subscribe("channel:1", "channel:2")
# Create reader task
reader_task = asyncio.create_task(reader(pubsub))
# Publish messages
await r.publish("channel:1", "Hello")
await r.publish("channel:2", "World")
await r.publish("channel:1", "STOP")
# Wait for reader to finish
await reader_task
await r.close()
# Run async example
asyncio.run(pubsub_example())
Sharded Pub/Sub (Redis 7.0+)
from redis.cluster import RedisCluster, ClusterNode
# Connect to cluster
rc = RedisCluster(startup_nodes=[
ClusterNode('localhost', 6379),
ClusterNode('localhost', 6380)
])
# Create sharded pubsub
p = rc.pubsub()
p.ssubscribe('foo')
# Get message from specific node
message = p.get_sharded_message(target_node=ClusterNode('localhost', 6379))
Distributed Locks
Simple Lock Implementation
import redis
import time
import uuid
class RedisLock:
"""Simple distributed lock using Redis."""
def __init__(self, redis_client: redis.Redis, key: str, timeout: int = 10):
self.redis = redis_client
self.key = f"lock:{key}"
self.timeout = timeout
self.identifier = str(uuid.uuid4())
def acquire(self, blocking: bool = True, timeout: float = None) -> bool:
"""Acquire lock."""
end_time = time.time() + (timeout or self.timeout)
while True:
# Try to set lock with NX (only if not exists) and EX (expiration)
if self.redis.set(self.key, self.identifier, nx=True, ex=self.timeout):
return True
if not blocking:
return False
if timeout and time.time() > end_time:
return False
# Wait before retry
time.sleep(0.01)
def release(self) -> bool:
"""Release lock only if we own it."""
# Use Lua script for atomic check-and-delete
lua_script = """
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
"""
result = self.redis.eval(lua_script, 1, self.key, self.identifier)
return result == 1
def __enter__(self):
"""Context manager support."""
self.acquire()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Context manager cleanup."""
self.release()
# Usage example
r = redis.Redis()
lock = RedisLock(r, "resource:123", timeout=5)
with lock:
# Critical section - only one process at a time
print("Processing resource 123")
process_resource()
Advanced Lock with Auto-Renewal
import threading
class RenewableLock:
"""Distributed lock with automatic renewal."""
def __init__(self, redis_client: redis.Redis, key: str, timeout: int = 10):
self.redis = redis_client
self.key = f"lock:{key}"
self.timeout = timeout
self.identifier = str(uuid.uuid4())
self.renewal_thread = None
self.stop_renewal = threading.Event()
def _renew_lock(self):
"""Background task to renew lock."""
while not self.stop_renewal.is_set():
time.sleep(self.timeout / 3) # Renew at 1/3 of timeout
# Renew only if we still own the lock
lua_script = """
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("expire", KEYS[1], ARGV[2])
else
return 0
end
"""
result = self.redis.eval(lua_script, 1, self.key,
self.identifier, self.timeout)
if result == 0:
# We lost the lock
self.stop_renewal.set()
def acquire(self, blocking: bool = True, timeout: float = None) -> bool:
"""Acquire lock and start auto-renewal."""
if self.redis.set(self.key, self.identifier, nx=True, ex=self.timeout):
# Start renewal thread
self.stop_renewal.clear()
self.renewal_thread = threading.Thread(target=self._renew_lock, daemon=True)
self.renewal_thread.start()
return True
return False
def release(self) -> bool:
"""Release lock and stop renewal."""
self.stop_renewal.set()
lua_script = """
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
"""
result = self.redis.eval(lua_script, 1, self.key, self.identifier)
return result == 1
Redlock Algorithm (Multiple Redis Instances)
class Redlock:
"""Redlock algorithm for distributed locking across multiple Redis instances."""
def __init__(self, redis_instances: list):
"""
Args:
redis_instances: List of Redis client connections
"""
self.instances = redis_instances
self.quorum = len(redis_instances) // 2 + 1
def acquire(self, resource: str, ttl: int = 10000) -> tuple:
"""
Acquire lock across multiple Redis instances.
Returns:
(success: bool, lock_identifier: str)
"""
identifier = str(uuid.uuid4())
start_time = int(time.time() * 1000)
# Try to acquire lock on all instances
acquired = 0
for instance in self.instances:
try:
if instance.set(f"lock:{resource}", identifier,
nx=True, px=ttl):
acquired += 1
except Exception:
pass
# Calculate elapsed time
elapsed = int(time.time() * 1000) - start_time
validity_time = ttl - elapsed - 100 # drift compensation
# Check if we got quorum
if acquired >= self.quorum and validity_time > 0:
return True, identifier
else:
# Release locks if we didn't get quorum
self._release_all(resource, identifier)
return False, None
def _release_all(self, resource: str, identifier: str):
"""Release lock on all instances."""
lua_script = """
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
"""
for instance in self.instances:
try:
instance.eval(lua_script, 1, f"lock:{resource}", identifier)
except Exception:
pass
Data Structures and Operations
Working with Hashes
# User profile storage
def save_user_profile(user_id: int, profile: dict):
"""Save user profile as hash."""
key = f"user:profile:{user_id}"
r.hset(key, mapping=profile)
r.expire(key, 86400) # 24 hour TTL
def get_user_profile(user_id: int) -> dict:
"""Get complete user profile."""
key = f"user:profile:{user_id}"
return r.hgetall(key)
def update_user_field(user_id: int, field: str, value: str):
"""Update single profile field."""
key = f"user:profile:{user_id}"
r.hset(key, field, value)
# Example usage
save_user_profile(123, {
"username": "alice",
"email": "alice@example.com",
"age": "30"
})
# Atomic increment
r.hincrby("user:profile:123", "login_count", 1)
Working with Lists
# Job queue implementation
def enqueue_job(queue_name: str, job_data: dict):
"""Add job to queue."""
key = f"queue:{queue_name}"
r.rpush(key, json.dumps(job_data))
def dequeue_job(queue_name: str, timeout: int = 0) -> dict:
"""Get job from queue (blocking)."""
key = f"queue:{queue_name}"
if timeout > 0:
# Blocking pop with timeout
result = r.blpop(key, timeout=timeout)
if result:
_, job_data = result
return json.loads(job_data)
else:
# Non-blocking pop
job_data = r.lpop(key)
if job_data:
return json.loads(job_data)
return None
# Activity feed
def add_to_feed(user_id: int, activity: dict):
"""Add activity to user feed."""
key = f"feed:{user_id}"
r.lpush(key, json.dumps(activity))
r.ltrim(key, 0, 99) # Keep only latest 100 items
r.expire(key, 604800) # 7 days
def get_feed(user_id: int, start: int = 0, end: int = 19) -> list:
"""Get user feed with pagination."""
key = f"feed:{user_id}"
items = r.lrange(key, start, end)
return [json.loads(item) for item in items]
Working with Sets
# Tags and relationships
def add_tags(item_id: int, tags: list):
"""Add tags to item."""
key = f"item:{item_id}:tags"
r.sadd(key, *tags)
def get_tags(item_id: int) -> set:
"""Get all tags for item."""
key = f"item:{item_id}:tags"
return r.smembers(key)
def find_items_with_all_tags(tags: list) -> set:
"""Find items having all specified tags."""
keys = [f"item:*:tags" for _ in tags]
# This is simplified - in practice, you'd need to track item IDs differently
return r.sinter(*keys)
# Online users tracking
def user_online(user_id: int):
"""Mark user as online."""
r.sadd("users:online", user_id)
r.expire(f"user:{user_id}:heartbeat", 60)
def user_offline(user_id: int):
"""Mark user as offline."""
r.srem("users:online", user_id)
def get_online_users() -> set:
"""Get all online users."""
return r.smembers("users:online")
def get_online_count() -> int:
"""Get count of online users."""
return r.scard("users:online")
Working with Sorted Sets
# Leaderboard implementation
def update_score(leaderboard: str, user_id: int, score: float):
"""Update user score in leaderboard."""
key = f"leaderboard:{leaderboard}"
r.zadd(key, {user_id: score})
def get_leaderboard(leaderboard: str, start: int = 0, end: int = 9) -> list:
"""Get top players (descending order)."""
key = f"leaderboard:{leaderboard}"
# ZREVRANGE for descending order (highest scores first)
return r.zrevrange(key, start, end, withscores=True)
def get_user_rank(leaderboard: str, user_id: int) -> int:
"""Get user's rank (0-indexed)."""
key = f"leaderboard:{leaderboard}"
# ZREVRANK for descending rank
rank = r.zrevrank(key, user_id)
return rank if rank is not None else -1
def get_user_score(leaderboard: str, user_id: int) -> float:
"""Get user's score."""
key = f"leaderboard:{leaderboard}"
score = r.zscore(key, user_id)
return score if score is not None else 0.0
def get_score_range(leaderboard: str, min_score: float, max_score: float) -> list:
"""Get users within score range."""
key = f"leaderboard:{leaderboard}"
return r.zrangebyscore(key, min_score, max_score, withscores=True)
# Time-based sorted set (activity stream)
def add_activity(user_id: int, activity: str):
"""Add timestamped activity."""
key = f"user:{user_id}:activities"
timestamp = time.time()
r.zadd(key, {activity: timestamp})
# Keep only last 24 hours
cutoff = timestamp - 86400
r.zremrangebyscore(key, '-inf', cutoff)
def get_recent_activities(user_id: int, count: int = 10) -> list:
"""Get recent activities."""
key = f"user:{user_id}:activities"
# Get most recent (highest timestamps)
return r.zrevrange(key, 0, count - 1, withscores=True)
Working with Streams
# Event stream
def add_event(stream_key: str, event_data: dict) -> str:
"""Add event to stream."""
# Returns auto-generated ID (timestamp-sequence)
event_id = r.xadd(stream_key, event_data)
return event_id
def read_events(stream_key: str, count: int = 10, start_id: str = '0') -> list:
"""Read events from stream."""
events = r.xread({stream_key: start_id}, count=count)
# events format: [(stream_name, [(id, data), (id, data), ...])]
if events:
_, event_list = events[0]
return event_list
return []
# Consumer groups
def create_consumer_group(stream_key: str, group_name: str):
"""Create consumer group for stream."""
try:
r.xgroup_create(name=stream_key, groupname=group_name, id='0')
except redis.ResponseError as e:
if "BUSYGROUP" not in str(e):
raise
def read_from_group(stream_key: str, group_name: str,
consumer_name: str, count: int = 10) -> list:
"""Read events as consumer in group."""
# Read new messages with '>'
events = r.xreadgroup(
groupname=group_name,
consumername=consumer_name,
streams={stream_key: '>'},
count=count,
block=5000 # 5 second timeout
)
if events:
_, event_list = events[0]
return event_list
return []
def acknowledge_event(stream_key: str, group_name: str, event_id: str):
"""Acknowledge processed event."""
r.xack(stream_key, group_name, event_id)
# Example: Processing events with consumer group
def process_events(stream_key: str, group_name: str, consumer_name: str):
"""Process events from stream."""
create_consumer_group(stream_key, group_name)
while True:
events = read_from_group(stream_key, group_name, consumer_name, count=10)
for event_id, event_data in events:
try:
# Process event
process_event(event_data)
# Acknowledge successful processing
acknowledge_event(stream_key, group_name, event_id)
except Exception as e:
print(f"Failed to process event {event_id}: {e}")
# Event remains unacknowledged for retry
Performance Optimization
Pipelining for Batch Operations
# Without pipelining (slow - multiple round trips)
for i in range(1000):
r.set(f"key:{i}", f"value:{i}")
# With pipelining (fast - single round trip)
pipe = r.pipeline()
for i in range(1000):
pipe.set(f"key:{i}", f"value:{i}")
results = pipe.execute()
# Pipelining with reads
pipe = r.pipeline()
for i in range(100):
pipe.get(f"key:{i}")
values = pipe.execute()
# Builder pattern with pipeline
class DataLoader:
def __init__(self):
self.pipeline = r.pipeline()
def add_user(self, user_id: int, user_data: dict):
"""Add user data."""
self.pipeline.hset(f"user:{user_id}", mapping=user_data)
return self
def add_to_set(self, set_name: str, value: str):
"""Add to set."""
self.pipeline.sadd(set_name, value)
return self
def execute(self):
"""Execute all pipelined commands."""
return self.pipeline.execute()
# Usage
loader = DataLoader()
results = (loader
.add_user(1, {"name": "Alice", "email": "alice@example.com"})
.add_user(2, {"name": "Bob", "email": "bob@example.com"})
.add_to_set("active_users", "1")
.add_to_set("active_users", "2")
.execute())
Transactions with WATCH
# Optimistic locking with WATCH
def transfer_credits(from_user: int, to_user: int, amount: int) -> bool:
"""Transfer credits between users with optimistic locking."""
with r.pipeline() as pipe:
while True:
try:
# Watch the keys we're going to modify
pipe.watch(f"user:{from_user}:credits", f"user:{to_user}:credits")
# Get current values
from_credits = int(pipe.get(f"user:{from_user}:credits") or 0)
to_credits = int(pipe.get(f"user:{to_user}:credits") or 0)
# Check if transfer is possible
if from_credits < amount:
pipe.unwatch()
return False
# Start transaction
pipe.multi()
pipe.set(f"user:{from_user}:credits", from_credits - amount)
pipe.set(f"user:{to_user}:credits", to_credits + amount)
# Execute transaction
pipe.execute()
return True
except redis.WatchError:
# Key was modified by another client - retry
continue
# Lua scripts for atomic operations
increment_script = """
local current = redis.call('GET', KEYS[1])
if not current then
current = 0
end
local new_val = tonumber(current) + tonumber(ARGV[1])
redis.call('SET', KEYS[1], new_val)
return new_val
"""
# Register and use Lua script
increment = r.register_script(increment_script)
new_value = increment(keys=['counter:views'], args=[1])
Lua Scripts for Complex Operations
# Rate limiting with Lua
rate_limit_script = """
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current = redis.call('INCR', key)
if current == 1 then
redis.call('EXPIRE', key, window)
end
if current > limit then
return 0
else
return 1
end
"""
rate_limiter = r.register_script(rate_limit_script)
def is_allowed(user_id: int, limit: int = 100, window: int = 60) -> bool:
"""Check if user is within rate limit."""
key = f"rate_limit:{user_id}"
result = rate_limiter(keys=[key], args=[limit, window])
return result == 1
# Get-or-set pattern with Lua
get_or_set_script = """
local value = redis.call('GET', KEYS[1])
if value then
return value
else
redis.call('SET', KEYS[1], ARGV[1])
redis.call('EXPIRE', KEYS[1], ARGV[2])
return ARGV[1]
end
"""
get_or_set = r.register_script(get_or_set_script)
def get_or_compute(key: str, compute_fn, ttl: int = 3600):
"""Get value from cache or compute and cache it."""
value = get_or_set(keys=[key], args=["__COMPUTING__", ttl])
if value == "__COMPUTING__":
# We set the placeholder - compute the real value
computed = compute_fn()
r.setex(key, ttl, computed)
return computed
return value
Production Patterns
High Availability with Sentinel
from redis.sentinel import Sentinel
# Connect to Sentinel
sentinel = Sentinel([
('sentinel1', 26379),
('sentinel2', 26379),
('sentinel3', 26379)
], socket_timeout=0.5)
# Get master connection
master = sentinel.master_for('mymaster', socket_timeout=0.5)
# Get replica connection (for read-only operations)
replica = sentinel.slave_for('mymaster', socket_timeout=0.5)
# Use master for writes
master.set('key', 'value')
# Use replica for reads (optional, for load distribution)
value = replica.get('key')
Async Redis with asyncio
import asyncio
import redis.asyncio as redis
async def async_redis_operations():
"""Async Redis operations example."""
# Create async connection
r = await redis.from_url("redis://localhost")
try:
# Async operations
await r.set("async_key", "async_value")
value = await r.get("async_key")
print(f"Value: {value}")
# Async pipeline
async with r.pipeline(transaction=True) as pipe:
await pipe.set("key1", "value1")
await pipe.set("key2", "value2")
await pipe.get("key1")
results = await pipe.execute()
print(f"Pipeline results: {results}")
finally:
await r.close()
# Run async operations
asyncio.run(async_redis_operations())
Connection Pool Configuration
# Production-ready connection pool
pool = redis.ConnectionPool(
host='localhost',
port=6379,
db=0,
max_connections=50, # Max pool size
socket_timeout=5, # Socket timeout
socket_connect_timeout=5, # Connection timeout
socket_keepalive=True, # Keep TCP connection alive
socket_keepalive_options={
socket.TCP_KEEPIDLE: 60,
socket.TCP_KEEPINTVL: 10,
socket.TCP_KEEPCNT: 3
},
retry_on_timeout=True, # Retry on timeout
health_check_interval=30, # Health check every 30s
decode_responses=True # Auto-decode bytes to strings
)
r = redis.Redis(connection_pool=pool)
Error Handling and Resilience
import redis
from redis.exceptions import ConnectionError, TimeoutError
import time
class ResilientRedisClient:
"""Redis client with retry logic and circuit breaker."""
def __init__(self, max_retries: int = 3, backoff: float = 0.1):
self.redis = redis.Redis(
host='localhost',
port=6379,
socket_timeout=5,
retry_on_timeout=True
)
self.max_retries = max_retries
self.backoff = backoff
def get_with_retry(self, key: str, default=None):
"""Get value with exponential backoff retry."""
for attempt in range(self.max_retries):
try:
return self.redis.get(key) or default
except (ConnectionError, TimeoutError) as e:
if attempt == self.max_retries - 1:
# Log error and return default
print(f"Redis error after {self.max_retries} attempts: {e}")
return default
# Exponential backoff
wait_time = self.backoff * (2 ** attempt)
time.sleep(wait_time)
def set_with_retry(self, key: str, value: str, ttl: int = None) -> bool:
"""Set value with retry logic."""
for attempt in range(self.max_retries):
try:
if ttl:
return self.redis.setex(key, ttl, value)
else:
return self.redis.set(key, value)
except (ConnectionError, TimeoutError) as e:
if attempt == self.max_retries - 1:
print(f"Redis error after {self.max_retries} attempts: {e}")
return False
wait_time = self.backoff * (2 ** attempt)
time.sleep(wait_time)
Monitoring and Metrics
def get_redis_info(section: str = None) -> dict:
"""Get Redis server information."""
return r.info(section=section)
def monitor_memory_usage():
"""Monitor Redis memory usage."""
info = r.info('memory')
used_memory = info['used_memory_human']
peak_memory = info['used_memory_peak_human']
memory_fragmentation = info['mem_fragmentation_ratio']
print(f"Used Memory: {used_memory}")
print(f"Peak Memory: {peak_memory}")
print(f"Fragmentation Ratio: {memory_fragmentation}")
return info
def monitor_stats():
"""Monitor Redis statistics."""
info = r.info('stats')
total_connections = info['total_connections_received']
total_commands = info['total_commands_processed']
ops_per_sec = info['instantaneous_ops_per_sec']
print(f"Total Connections: {total_connections}")
print(f"Total Commands: {total_commands}")
print(f"Ops/sec: {ops_per_sec}")
return info
def get_slow_log(count: int = 10):
"""Get slow query log."""
slow_log = r.slowlog_get(count)
for entry in slow_log:
print(f"Command: {entry['command']}")
…(truncated)
1---
2name: redis-state-management
3description: Comprehensive guide for Redis state management including caching strategies, session management, pub/sub patterns, distributed locks, and data structures
4tags: [redis, state-management, caching, pub-sub, distributed-systems, sessions]
5tier: tier-1
6metadata:
7 source: https://skillmd.com/skills/manutej/redis-state-management
8---
9
10# Redis State Management
11
12A comprehensive skill for mastering Redis state management patterns in distributed systems. This skill covers caching strategies, session management, pub/sub messaging, distributed locks, data structures, and production-ready patterns using redis-py.
13
14## When to Use This Skill
15
16Use this skill when:
17
18- Implementing high-performance caching layers for web applications
19- Managing user sessions in distributed environments
20- Building real-time messaging and event distribution systems
21- Coordinating distributed processes with locks and synchronization
22- Storing and querying structured data with Redis data structures
23- Optimizing application performance with Redis
24- Scaling applications horizontally with shared state
25- Implementing rate limiting, counters, and analytics
26- Building microservices with Redis as a communication layer
27- Managing temporary data with automatic expiration (TTL)
28- Implementing leaderboards, queues, and real-time features
29
30## Core Concepts
31
32### Redis Fundamentals
33
34**Redis** (Remote Dictionary Server) is an in-memory data structure store used as:
35- **Database**: Persistent key-value storage
36- **Cache**: High-speed data layer
37- **Message Broker**: Pub/sub and stream messaging
38- **Session Store**: Distributed session management
39
40**Key Characteristics:**
41- In-memory storage (microsecond latency)
42- Optional persistence (RDB snapshots, AOF logs)
43- Rich data structures beyond key-value
44- Atomic operations on complex data types
45- Built-in replication and clustering
46- Pub/sub messaging support
47- Lua scripting for complex operations
48- Pipelining for batch operations
49
50### Redis Data Structures
51
52Redis provides multiple data types for different use cases:
53
541. **Strings**: Simple key-value pairs, binary safe
55 - Use for: Cache values, counters, flags, JSON objects
56 - Max size: 512 MB
57 - Commands: SET, GET, INCR, APPEND
58
592. **Hashes**: Field-value maps (objects)
60 - Use for: User profiles, configuration objects, small entities
61 - Efficient for storing objects with multiple fields
62 - Commands: HSET, HGET, HMGET, HINCRBY
63
643. **Lists**: Ordered collections (linked lists)
65 - Use for: Queues, activity feeds, recent items
66 - Operations at head/tail are O(1)
67 - Commands: LPUSH, RPUSH, LPOP, RPOP, LRANGE
68
694. **Sets**: Unordered unique collections
70 - Use for: Tags, unique visitors, relationships
71 - Set operations: union, intersection, difference
72 - Commands: SADD, SMEMBERS, SISMEMBER, SINTER
73
745. **Sorted Sets**: Ordered sets with scores
75 - Use for: Leaderboards, time-series, priority queues
76 - Range queries by score or rank
77 - Commands: ZADD, ZRANGE, ZRANGEBYSCORE, ZRANK
78
796. **Streams**: Append-only logs with consumer groups
80 - Use for: Event sourcing, activity logs, message queues
81 - Built-in consumer group support
82 - Commands: XADD, XREAD, XREADGROUP
83
84### Connection Management
85
86**Connection Pools:**
87Redis connections are expensive to create. Always use connection pools:
88
89```python
90import redis
91
92# Connection pool (recommended)
93pool = redis.ConnectionPool(host='localhost', port=6379, db=0, max_connections=10)
94r = redis.Redis(connection_pool=pool)
95
96# Direct connection (avoid in production)
97r = redis.Redis(host='localhost', port=6379, db=0)
98```
99
100**Best Practices:**
101- Use connection pools for all applications
102- Set appropriate max_connections based on workload
103- Enable decode_responses=True for string data
104- Configure socket_timeout and socket_keepalive
105- Handle connection errors with retries
106
107### Data Persistence
108
109Redis offers two persistence mechanisms:
110
111**RDB (Redis Database)**: Point-in-time snapshots
112- Compact binary format
113- Fast restart times
114- Lower disk I/O
115- Potential data loss between snapshots
116
117**AOF (Append-Only File)**: Log of write operations
118- Better durability (fsync policies)
119- Larger files, slower restarts
120- Can be automatically rewritten/compacted
121- Minimal data loss potential
122
123**Hybrid Approach**: RDB + AOF for best of both worlds
124
125### RESP 3 Protocol
126
127Redis Serialization Protocol version 3 offers:
128- Client-side caching support
129- Better data type support
130- Push notifications
131- Performance improvements
132
133```python
134import redis
135from redis.cache import CacheConfig
136
137# Enable RESP3 with client-side caching
138r = redis.Redis(host='localhost', port=6379, protocol=3,
139 cache_config=CacheConfig())
140```
141
142## Caching Strategies
143
144### Cache-Aside (Lazy Loading)
145
146**Pattern**: Application checks cache first, loads from database on miss
147
148```python
149import redis
150import json
151from typing import Optional, Dict, Any
152
153r = redis.Redis(decode_responses=True)
154
155def get_user(user_id: int) -> Optional[Dict[str, Any]]:
156 """Cache-aside pattern for user data."""
157 cache_key = f"user:{user_id}"
158
159 # Try cache first
160 cached_data = r.get(cache_key)
161 if cached_data:
162 return json.loads(cached_data)
163
164 # Cache miss - load from database
165 user_data = database.get_user(user_id) # Your DB query
166 if user_data:
167 # Store in cache with 1 hour TTL
168 r.setex(cache_key, 3600, json.dumps(user_data))
169
170 return user_data
171```
172
173**Advantages:**
174- Only requested data is cached (efficient memory usage)
175- Cache failures don't break the application
176- Simple to implement
177
178**Disadvantages:**
179- Cache miss penalty (latency spike)
180- Thundering herd on popular items
181- Stale data until cache expiration
182
183### Write-Through Cache
184
185**Pattern**: Write to cache and database simultaneously
186
187```python
188def update_user(user_id: int, user_data: Dict[str, Any]) -> bool:
189 """Write-through pattern for user updates."""
190 cache_key = f"user:{user_id}"
191
192 # Write to database first
193 success = database.update_user(user_id, user_data)
194
195 if success:
196 # Update cache immediately
197 r.setex(cache_key, 3600, json.dumps(user_data))
198
199 return success
200```
201
202**Advantages:**
203- Cache always consistent with database
204- No read penalty for recently written data
205
206**Disadvantages:**
207- Write latency increases
208- Unused data may be cached
209- Extra cache write overhead
210
211### Write-Behind (Write-Back) Cache
212
213**Pattern**: Write to cache immediately, sync to database asynchronously
214
215```python
216import redis
217import json
218from queue import Queue
219from threading import Thread
220
221r = redis.Redis(decode_responses=True)
222write_queue = Queue()
223
224def async_writer():
225 """Background worker to sync cache to database."""
226 while True:
227 user_id, user_data = write_queue.get()
228 try:
229 database.update_user(user_id, user_data)
230 except Exception as e:
231 # Log error, potentially retry
232 print(f"Failed to write user {user_id}: {e}")
233 finally:
234 write_queue.task_done()
235
236# Start background writer
237Thread(target=async_writer, daemon=True).start()
238
239def update_user_fast(user_id: int, user_data: Dict[str, Any]):
240 """Write-behind pattern for fast writes."""
241 cache_key = f"user:{user_id}"
242
243 # Write to cache immediately (fast)
244 r.setex(cache_key, 3600, json.dumps(user_data))
245
246 # Queue database write (async)
247 write_queue.put((user_id, user_data))
248```
249
250**Advantages:**
251- Minimal write latency
252- Can batch database writes
253- Handles write spikes
254
255**Disadvantages:**
256- Risk of data loss if cache fails
257- Complex error handling
258- Consistency challenges
259
260### Cache Invalidation Strategies
261
262**Time-based Expiration (TTL):**
263
264```python
265# Set key with expiration
266r.setex("session:abc123", 1800, session_data) # 30 minutes
267
268# Or set TTL on existing key
269r.expire("user:profile:123", 3600) # 1 hour
270
271# Check remaining TTL
272ttl = r.ttl("user:profile:123")
273```
274
275**Event-based Invalidation:**
276
277```python
278def update_product(product_id: int, product_data: dict):
279 """Invalidate cache on update."""
280 # Update database
281 database.update_product(product_id, product_data)
282
283 # Invalidate related caches
284 r.delete(f"product:{product_id}")
285 r.delete(f"product_list:category:{product_data['category']}")
286 r.delete("products:featured")
287```
288
289**Pattern-based Invalidation:**
290
291```python
292# Delete all keys matching pattern
293def invalidate_user_cache(user_id: int):
294 """Invalidate all cache entries for a user."""
295 pattern = f"user:{user_id}:*"
296
297 # Find and delete matching keys
298 for key in r.scan_iter(match=pattern, count=100):
299 r.delete(key)
300```
301
302### Cache Stampede Prevention
303
304**Problem**: Multiple requests simultaneously miss cache and query database
305
306**Solution 1: Probabilistic Early Expiration**
307
308```python
309import time
310import random
311
312def get_with_early_expiration(key: str, ttl: int = 3600, beta: float = 1.0):
313 """Prevent stampede with probabilistic early recomputation."""
314 value = r.get(key)
315
316 if value is None:
317 # Cache miss - compute and cache
318 value = compute_value(key)
319 r.setex(key, ttl, value)
320 return value
321
322 # Check if we should recompute early
323 current_time = time.time()
324 delta = current_time - float(r.get(f"{key}:timestamp") or 0)
325 expiry = ttl * random.random() * beta
326
327 if delta > expiry:
328 # Recompute in background
329 value = compute_value(key)
330 r.setex(key, ttl, value)
331 r.set(f"{key}:timestamp", current_time)
332
333 return value
334```
335
336**Solution 2: Locking**
337
338```python
339from contextlib import contextmanager
340
341@contextmanager
342def cache_lock(key: str, timeout: int = 10):
343 """Acquire lock for cache computation."""
344 lock_key = f"{key}:lock"
345 identifier = str(time.time())
346
347 # Try to acquire lock
348 if r.set(lock_key, identifier, nx=True, ex=timeout):
349 try:
350 yield True
351 finally:
352 # Release lock
353 if r.get(lock_key) == identifier:
354 r.delete(lock_key)
355 else:
356 yield False
357
358def get_with_lock(key: str):
359 """Use lock to prevent stampede."""
360 value = r.get(key)
361
362 if value is None:
363 with cache_lock(key) as acquired:
364 if acquired:
365 # We got the lock - compute value
366 value = compute_value(key)
367 r.setex(key, 3600, value)
368 else:
369 # Someone else is computing - wait and retry
370 time.sleep(0.1)
371 value = r.get(key) or compute_value(key)
372
373 return value
374```
375
376## Session Management
377
378### Distributed Session Storage
379
380**Basic Session Management:**
381
382```python
383import redis
384import json
385import uuid
386from datetime import datetime, timedelta
387
388r = redis.Redis(decode_responses=True)
389
390class SessionManager:
391 def __init__(self, ttl: int = 1800):
392 """Session manager with Redis backend.
393
394 Args:
395 ttl: Session timeout in seconds (default 30 minutes)
396 """
397 self.ttl = ttl
398
399 def create_session(self, user_id: int, data: dict = None) -> str:
400 """Create new session and return session ID."""
401 session_id = str(uuid.uuid4())
402 session_key = f"session:{session_id}"
403
404 session_data = {
405 "user_id": user_id,
406 "created_at": datetime.utcnow().isoformat(),
407 "data": data or {}
408 }
409
410 r.setex(session_key, self.ttl, json.dumps(session_data))
411 return session_id
412
413 def get_session(self, session_id: str) -> dict:
414 """Retrieve session data and refresh TTL."""
415 session_key = f"session:{session_id}"
416 session_data = r.get(session_key)
417
418 if session_data:
419 # Refresh TTL on access (sliding expiration)
420 r.expire(session_key, self.ttl)
421 return json.loads(session_data)
422
423 return None
424
425 def update_session(self, session_id: str, data: dict) -> bool:
426 """Update session data."""
427 session_key = f"session:{session_id}"
428 session_data = self.get_session(session_id)
429
430 if session_data:
431 session_data["data"].update(data)
432 r.setex(session_key, self.ttl, json.dumps(session_data))
433 return True
434
435 return False
436
437 def delete_session(self, session_id: str) -> bool:
438 """Delete session (logout)."""
439 session_key = f"session:{session_id}"
440 return r.delete(session_key) > 0
441```
442
443### Session with Hash Storage
444
445**More efficient for session objects:**
446
447```python
448class HashSessionManager:
449 """Session manager using Redis hashes for better performance."""
450
451 def __init__(self, ttl: int = 1800):
452 self.ttl = ttl
453
454 def create_session(self, user_id: int, **kwargs) -> str:
455 """Create session using hash."""
456 session_id = str(uuid.uuid4())
457 session_key = f"session:{session_id}"
458
459 # Store as hash for efficient field access
460 session_fields = {
461 "user_id": str(user_id),
462 "created_at": datetime.utcnow().isoformat(),
463 **{k: str(v) for k, v in kwargs.items()}
464 }
465
466 r.hset(session_key, mapping=session_fields)
467 r.expire(session_key, self.ttl)
468
469 return session_id
470
471 def get_field(self, session_id: str, field: str) -> str:
472 """Get single session field efficiently."""
473 session_key = f"session:{session_id}"
474 value = r.hget(session_key, field)
475
476 if value:
477 r.expire(session_key, self.ttl) # Refresh TTL
478
479 return value
480
481 def set_field(self, session_id: str, field: str, value: str) -> bool:
482 """Update single session field."""
483 session_key = f"session:{session_id}"
484
485 if r.exists(session_key):
486 r.hset(session_key, field, value)
487 r.expire(session_key, self.ttl)
488 return True
489
490 return False
491
492 def get_all(self, session_id: str) -> dict:
493 """Get all session fields."""
494 session_key = f"session:{session_id}"
495 data = r.hgetall(session_key)
496
497 if data:
498 r.expire(session_key, self.ttl)
499
500 return data
501```
502
503### User Activity Tracking
504
505```python
506def track_user_activity(user_id: int, action: str):
507 """Track user activity with automatic expiration."""
508 activity_key = f"user:{user_id}:activity"
509 timestamp = datetime.utcnow().isoformat()
510
511 # Add activity to list
512 r.lpush(activity_key, json.dumps({"action": action, "timestamp": timestamp}))
513
514 # Keep only last 100 activities
515 r.ltrim(activity_key, 0, 99)
516
517 # Set expiration (30 days)
518 r.expire(activity_key, 2592000)
519
520def get_recent_activity(user_id: int, limit: int = 10) -> list:
521 """Get recent user activities."""
522 activity_key = f"user:{user_id}:activity"
523 activities = r.lrange(activity_key, 0, limit - 1)
524
525 return [json.loads(a) for a in activities]
526```
527
528## Pub/Sub Patterns
529
530### Basic Publisher/Subscriber
531
532**Publisher:**
533
534```python
535import redis
536
537r = redis.Redis(decode_responses=True)
538
539def publish_event(channel: str, message: dict):
540 """Publish event to channel."""
541 import json
542 r.publish(channel, json.dumps(message))
543
544# Example usage
545publish_event("notifications", {
546 "type": "user_signup",
547 "user_id": 12345,
548 "timestamp": datetime.utcnow().isoformat()
549})
550```
551
552**Subscriber:**
553
554```python
555import redis
556import json
557
558def handle_message(message):
559 """Process received message."""
560 data = json.loads(message['data'])
561 print(f"Received: {data}")
562
563# Initialize pubsub
564r = redis.Redis(decode_responses=True)
565p = r.pubsub()
566
567# Subscribe to channels
568p.subscribe('notifications', 'alerts')
569
570# Listen for messages
571for message in p.listen():
572 if message['type'] == 'message':
573 handle_message(message)
574```
575
576### Pattern-Based Subscriptions
577
578```python
579# Subscribe to multiple channels with patterns
580p = r.pubsub()
581p.psubscribe('user:*', 'notification:*')
582
583# Get messages from pattern subscriptions
584for message in p.listen():
585 if message['type'] == 'pmessage':
586 channel = message['channel']
587 pattern = message['pattern']
588 data = message['data']
589 print(f"Pattern {pattern} matched {channel}: {data}")
590```
591
592### Async Pub/Sub with Background Thread
593
594```python
595import redis
596import time
597
598r = redis.Redis(decode_responses=True)
599p = r.pubsub()
600
601def message_handler(message):
602 """Handle messages in background thread."""
603 print(f"Handler received: {message['data']}")
604
605# Subscribe with handler
606p.subscribe(**{'notifications': message_handler, 'alerts': message_handler})
607
608# Run in background thread
609thread = p.run_in_thread(sleep_time=0.001)
610
611# Publish some messages
612r.publish('notifications', 'Hello!')
613r.publish('alerts', 'Warning!')
614
615time.sleep(1)
616
617# Stop background thread
618thread.stop()
619```
620
621### Async Pub/Sub with asyncio
622
623```python
624import asyncio
625import redis.asyncio as redis
626
627async def reader(channel: redis.client.PubSub):
628 """Async message reader."""
629 while True:
630 message = await channel.get_message(ignore_subscribe_messages=True, timeout=None)
631 if message is not None:
632 print(f"Received: {message}")
633
634 # Stop on specific message
635 if message["data"].decode() == "STOP":
636 break
637
638async def pubsub_example():
639 """Async pub/sub example."""
640 r = await redis.from_url("redis://localhost")
641
642 async with r.pubsub() as pubsub:
643 # Subscribe to channels
644 await pubsub.subscribe("channel:1", "channel:2")
645
646 # Create reader task
647 reader_task = asyncio.create_task(reader(pubsub))
648
649 # Publish messages
650 await r.publish("channel:1", "Hello")
651 await r.publish("channel:2", "World")
652 await r.publish("channel:1", "STOP")
653
654 # Wait for reader to finish
655 await reader_task
656
657 await r.close()
658
659# Run async example
660asyncio.run(pubsub_example())
661```
662
663### Sharded Pub/Sub (Redis 7.0+)
664
665```python
666from redis.cluster import RedisCluster, ClusterNode
667
668# Connect to cluster
669rc = RedisCluster(startup_nodes=[
670 ClusterNode('localhost', 6379),
671 ClusterNode('localhost', 6380)
672])
673
674# Create sharded pubsub
675p = rc.pubsub()
676p.ssubscribe('foo')
677
678# Get message from specific node
679message = p.get_sharded_message(target_node=ClusterNode('localhost', 6379))
680```
681
682## Distributed Locks
683
684### Simple Lock Implementation
685
686```python
687import redis
688import time
689import uuid
690
691class RedisLock:
692 """Simple distributed lock using Redis."""
693
694 def __init__(self, redis_client: redis.Redis, key: str, timeout: int = 10):
695 self.redis = redis_client
696 self.key = f"lock:{key}"
697 self.timeout = timeout
698 self.identifier = str(uuid.uuid4())
699
700 def acquire(self, blocking: bool = True, timeout: float = None) -> bool:
701 """Acquire lock."""
702 end_time = time.time() + (timeout or self.timeout)
703
704 while True:
705 # Try to set lock with NX (only if not exists) and EX (expiration)
706 if self.redis.set(self.key, self.identifier, nx=True, ex=self.timeout):
707 return True
708
709 if not blocking:
710 return False
711
712 if timeout and time.time() > end_time:
713 return False
714
715 # Wait before retry
716 time.sleep(0.01)
717
718 def release(self) -> bool:
719 """Release lock only if we own it."""
720 # Use Lua script for atomic check-and-delete
721 lua_script = """
722 if redis.call("get", KEYS[1]) == ARGV[1] then
723 return redis.call("del", KEYS[1])
724 else
725 return 0
726 end
727 """
728
729 result = self.redis.eval(lua_script, 1, self.key, self.identifier)
730 return result == 1
731
732 def __enter__(self):
733 """Context manager support."""
734 self.acquire()
735 return self
736
737 def __exit__(self, exc_type, exc_val, exc_tb):
738 """Context manager cleanup."""
739 self.release()
740
741# Usage example
742r = redis.Redis()
743lock = RedisLock(r, "resource:123", timeout=5)
744
745with lock:
746 # Critical section - only one process at a time
747 print("Processing resource 123")
748 process_resource()
749```
750
751### Advanced Lock with Auto-Renewal
752
753```python
754import threading
755
756class RenewableLock:
757 """Distributed lock with automatic renewal."""
758
759 def __init__(self, redis_client: redis.Redis, key: str, timeout: int = 10):
760 self.redis = redis_client
761 self.key = f"lock:{key}"
762 self.timeout = timeout
763 self.identifier = str(uuid.uuid4())
764 self.renewal_thread = None
765 self.stop_renewal = threading.Event()
766
767 def _renew_lock(self):
768 """Background task to renew lock."""
769 while not self.stop_renewal.is_set():
770 time.sleep(self.timeout / 3) # Renew at 1/3 of timeout
771
772 # Renew only if we still own the lock
773 lua_script = """
774 if redis.call("get", KEYS[1]) == ARGV[1] then
775 return redis.call("expire", KEYS[1], ARGV[2])
776 else
777 return 0
778 end
779 """
780
781 result = self.redis.eval(lua_script, 1, self.key,
782 self.identifier, self.timeout)
783
784 if result == 0:
785 # We lost the lock
786 self.stop_renewal.set()
787
788 def acquire(self, blocking: bool = True, timeout: float = None) -> bool:
789 """Acquire lock and start auto-renewal."""
790 if self.redis.set(self.key, self.identifier, nx=True, ex=self.timeout):
791 # Start renewal thread
792 self.stop_renewal.clear()
793 self.renewal_thread = threading.Thread(target=self._renew_lock, daemon=True)
794 self.renewal_thread.start()
795 return True
796
797 return False
798
799 def release(self) -> bool:
800 """Release lock and stop renewal."""
801 self.stop_renewal.set()
802
803 lua_script = """
804 if redis.call("get", KEYS[1]) == ARGV[1] then
805 return redis.call("del", KEYS[1])
806 else
807 return 0
808 end
809 """
810
811 result = self.redis.eval(lua_script, 1, self.key, self.identifier)
812 return result == 1
813```
814
815### Redlock Algorithm (Multiple Redis Instances)
816
817```python
818class Redlock:
819 """Redlock algorithm for distributed locking across multiple Redis instances."""
820
821 def __init__(self, redis_instances: list):
822 """
823 Args:
824 redis_instances: List of Redis client connections
825 """
826 self.instances = redis_instances
827 self.quorum = len(redis_instances) // 2 + 1
828
829 def acquire(self, resource: str, ttl: int = 10000) -> tuple:
830 """
831 Acquire lock across multiple Redis instances.
832
833 Returns:
834 (success: bool, lock_identifier: str)
835 """
836 identifier = str(uuid.uuid4())
837 start_time = int(time.time() * 1000)
838
839 # Try to acquire lock on all instances
840 acquired = 0
841 for instance in self.instances:
842 try:
843 if instance.set(f"lock:{resource}", identifier,
844 nx=True, px=ttl):
845 acquired += 1
846 except Exception:
847 pass
848
849 # Calculate elapsed time
850 elapsed = int(time.time() * 1000) - start_time
851 validity_time = ttl - elapsed - 100 # drift compensation
852
853 # Check if we got quorum
854 if acquired >= self.quorum and validity_time > 0:
855 return True, identifier
856 else:
857 # Release locks if we didn't get quorum
858 self._release_all(resource, identifier)
859 return False, None
860
861 def _release_all(self, resource: str, identifier: str):
862 """Release lock on all instances."""
863 lua_script = """
864 if redis.call("get", KEYS[1]) == ARGV[1] then
865 return redis.call("del", KEYS[1])
866 else
867 return 0
868 end
869 """
870
871 for instance in self.instances:
872 try:
873 instance.eval(lua_script, 1, f"lock:{resource}", identifier)
874 except Exception:
875 pass
876```
877
878## Data Structures and Operations
879
880### Working with Hashes
881
882```python
883# User profile storage
884def save_user_profile(user_id: int, profile: dict):
885 """Save user profile as hash."""
886 key = f"user:profile:{user_id}"
887 r.hset(key, mapping=profile)
888 r.expire(key, 86400) # 24 hour TTL
889
890def get_user_profile(user_id: int) -> dict:
891 """Get complete user profile."""
892 key = f"user:profile:{user_id}"
893 return r.hgetall(key)
894
895def update_user_field(user_id: int, field: str, value: str):
896 """Update single profile field."""
897 key = f"user:profile:{user_id}"
898 r.hset(key, field, value)
899
900# Example usage
901save_user_profile(123, {
902 "username": "alice",
903 "email": "alice@example.com",
904 "age": "30"
905})
906
907# Atomic increment
908r.hincrby("user:profile:123", "login_count", 1)
909```
910
911### Working with Lists
912
913```python
914# Job queue implementation
915def enqueue_job(queue_name: str, job_data: dict):
916 """Add job to queue."""
917 key = f"queue:{queue_name}"
918 r.rpush(key, json.dumps(job_data))
919
920def dequeue_job(queue_name: str, timeout: int = 0) -> dict:
921 """Get job from queue (blocking)."""
922 key = f"queue:{queue_name}"
923
924 if timeout > 0:
925 # Blocking pop with timeout
926 result = r.blpop(key, timeout=timeout)
927 if result:
928 _, job_data = result
929 return json.loads(job_data)
930 else:
931 # Non-blocking pop
932 job_data = r.lpop(key)
933 if job_data:
934 return json.loads(job_data)
935
936 return None
937
938# Activity feed
939def add_to_feed(user_id: int, activity: dict):
940 """Add activity to user feed."""
941 key = f"feed:{user_id}"
942 r.lpush(key, json.dumps(activity))
943 r.ltrim(key, 0, 99) # Keep only latest 100 items
944 r.expire(key, 604800) # 7 days
945
946def get_feed(user_id: int, start: int = 0, end: int = 19) -> list:
947 """Get user feed with pagination."""
948 key = f"feed:{user_id}"
949 items = r.lrange(key, start, end)
950 return [json.loads(item) for item in items]
951```
952
953### Working with Sets
954
955```python
956# Tags and relationships
957def add_tags(item_id: int, tags: list):
958 """Add tags to item."""
959 key = f"item:{item_id}:tags"
960 r.sadd(key, *tags)
961
962def get_tags(item_id: int) -> set:
963 """Get all tags for item."""
964 key = f"item:{item_id}:tags"
965 return r.smembers(key)
966
967def find_items_with_all_tags(tags: list) -> set:
968 """Find items having all specified tags."""
969 keys = [f"item:*:tags" for _ in tags]
970 # This is simplified - in practice, you'd need to track item IDs differently
971 return r.sinter(*keys)
972
973# Online users tracking
974def user_online(user_id: int):
975 """Mark user as online."""
976 r.sadd("users:online", user_id)
977 r.expire(f"user:{user_id}:heartbeat", 60)
978
979def user_offline(user_id: int):
980 """Mark user as offline."""
981 r.srem("users:online", user_id)
982
983def get_online_users() -> set:
984 """Get all online users."""
985 return r.smembers("users:online")
986
987def get_online_count() -> int:
988 """Get count of online users."""
989 return r.scard("users:online")
990```
991
992### Working with Sorted Sets
993
994```python
995# Leaderboard implementation
996def update_score(leaderboard: str, user_id: int, score: float):
997 """Update user score in leaderboard."""
998 key = f"leaderboard:{leaderboard}"
999 r.zadd(key, {user_id: score})
1000
1001def get_leaderboard(leaderboard: str, start: int = 0, end: int = 9) -> list:
1002 """Get top players (descending order)."""
1003 key = f"leaderboard:{leaderboard}"
1004 # ZREVRANGE for descending order (highest scores first)
1005 return r.zrevrange(key, start, end, withscores=True)
1006
1007def get_user_rank(leaderboard: str, user_id: int) -> int:
1008 """Get user's rank (0-indexed)."""
1009 key = f"leaderboard:{leaderboard}"
1010 # ZREVRANK for descending rank
1011 rank = r.zrevrank(key, user_id)
1012 return rank if rank is not None else -1
1013
1014def get_user_score(leaderboard: str, user_id: int) -> float:
1015 """Get user's score."""
1016 key = f"leaderboard:{leaderboard}"
1017 score = r.zscore(key, user_id)
1018 return score if score is not None else 0.0
1019
1020def get_score_range(leaderboard: str, min_score: float, max_score: float) -> list:
1021 """Get users within score range."""
1022 key = f"leaderboard:{leaderboard}"
1023 return r.zrangebyscore(key, min_score, max_score, withscores=True)
1024
1025# Time-based sorted set (activity stream)
1026def add_activity(user_id: int, activity: str):
1027 """Add timestamped activity."""
1028 key = f"user:{user_id}:activities"
1029 timestamp = time.time()
1030 r.zadd(key, {activity: timestamp})
1031
1032 # Keep only last 24 hours
1033 cutoff = timestamp - 86400
1034 r.zremrangebyscore(key, '-inf', cutoff)
1035
1036def get_recent_activities(user_id: int, count: int = 10) -> list:
1037 """Get recent activities."""
1038 key = f"user:{user_id}:activities"
1039 # Get most recent (highest timestamps)
1040 return r.zrevrange(key, 0, count - 1, withscores=True)
1041```
1042
1043### Working with Streams
1044
1045```python
1046# Event stream
1047def add_event(stream_key: str, event_data: dict) -> str:
1048 """Add event to stream."""
1049 # Returns auto-generated ID (timestamp-sequence)
1050 event_id = r.xadd(stream_key, event_data)
1051 return event_id
1052
1053def read_events(stream_key: str, count: int = 10, start_id: str = '0') -> list:
1054 """Read events from stream."""
1055 events = r.xread({stream_key: start_id}, count=count)
1056
1057 # events format: [(stream_name, [(id, data), (id, data), ...])]
1058 if events:
1059 _, event_list = events[0]
1060 return event_list
1061
1062 return []
1063
1064# Consumer groups
1065def create_consumer_group(stream_key: str, group_name: str):
1066 """Create consumer group for stream."""
1067 try:
1068 r.xgroup_create(name=stream_key, groupname=group_name, id='0')
1069 except redis.ResponseError as e:
1070 if "BUSYGROUP" not in str(e):
1071 raise
1072
1073def read_from_group(stream_key: str, group_name: str,
1074 consumer_name: str, count: int = 10) -> list:
1075 """Read events as consumer in group."""
1076 # Read new messages with '>'
1077 events = r.xreadgroup(
1078 groupname=group_name,
1079 consumername=consumer_name,
1080 streams={stream_key: '>'},
1081 count=count,
1082 block=5000 # 5 second timeout
1083 )
1084
1085 if events:
1086 _, event_list = events[0]
1087 return event_list
1088
1089 return []
1090
1091def acknowledge_event(stream_key: str, group_name: str, event_id: str):
1092 """Acknowledge processed event."""
1093 r.xack(stream_key, group_name, event_id)
1094
1095# Example: Processing events with consumer group
1096def process_events(stream_key: str, group_name: str, consumer_name: str):
1097 """Process events from stream."""
1098 create_consumer_group(stream_key, group_name)
1099
1100 while True:
1101 events = read_from_group(stream_key, group_name, consumer_name, count=10)
1102
1103 for event_id, event_data in events:
1104 try:
1105 # Process event
1106 process_event(event_data)
1107
1108 # Acknowledge successful processing
1109 acknowledge_event(stream_key, group_name, event_id)
1110 except Exception as e:
1111 print(f"Failed to process event {event_id}: {e}")
1112 # Event remains unacknowledged for retry
1113```
1114
1115## Performance Optimization
1116
1117### Pipelining for Batch Operations
1118
1119```python
1120# Without pipelining (slow - multiple round trips)
1121for i in range(1000):
1122 r.set(f"key:{i}", f"value:{i}")
1123
1124# With pipelining (fast - single round trip)
1125pipe = r.pipeline()
1126for i in range(1000):
1127 pipe.set(f"key:{i}", f"value:{i}")
1128results = pipe.execute()
1129
1130# Pipelining with reads
1131pipe = r.pipeline()
1132for i in range(100):
1133 pipe.get(f"key:{i}")
1134values = pipe.execute()
1135
1136# Builder pattern with pipeline
1137class DataLoader:
1138 def __init__(self):
1139 self.pipeline = r.pipeline()
1140
1141 def add_user(self, user_id: int, user_data: dict):
1142 """Add user data."""
1143 self.pipeline.hset(f"user:{user_id}", mapping=user_data)
1144 return self
1145
1146 def add_to_set(self, set_name: str, value: str):
1147 """Add to set."""
1148 self.pipeline.sadd(set_name, value)
1149 return self
1150
1151 def execute(self):
1152 """Execute all pipelined commands."""
1153 return self.pipeline.execute()
1154
1155# Usage
1156loader = DataLoader()
1157results = (loader
1158 .add_user(1, {"name": "Alice", "email": "alice@example.com"})
1159 .add_user(2, {"name": "Bob", "email": "bob@example.com"})
1160 .add_to_set("active_users", "1")
1161 .add_to_set("active_users", "2")
1162 .execute())
1163```
1164
1165### Transactions with WATCH
1166
1167```python
1168# Optimistic locking with WATCH
1169def transfer_credits(from_user: int, to_user: int, amount: int) -> bool:
1170 """Transfer credits between users with optimistic locking."""
1171
1172 with r.pipeline() as pipe:
1173 while True:
1174 try:
1175 # Watch the keys we're going to modify
1176 pipe.watch(f"user:{from_user}:credits", f"user:{to_user}:credits")
1177
1178 # Get current values
1179 from_credits = int(pipe.get(f"user:{from_user}:credits") or 0)
1180 to_credits = int(pipe.get(f"user:{to_user}:credits") or 0)
1181
1182 # Check if transfer is possible
1183 if from_credits < amount:
1184 pipe.unwatch()
1185 return False
1186
1187 # Start transaction
1188 pipe.multi()
1189 pipe.set(f"user:{from_user}:credits", from_credits - amount)
1190 pipe.set(f"user:{to_user}:credits", to_credits + amount)
1191
1192 # Execute transaction
1193 pipe.execute()
1194 return True
1195
1196 except redis.WatchError:
1197 # Key was modified by another client - retry
1198 continue
1199
1200# Lua scripts for atomic operations
1201increment_script = """
1202local current = redis.call('GET', KEYS[1])
1203if not current then
1204 current = 0
1205end
1206local new_val = tonumber(current) + tonumber(ARGV[1])
1207redis.call('SET', KEYS[1], new_val)
1208return new_val
1209"""
1210
1211# Register and use Lua script
1212increment = r.register_script(increment_script)
1213new_value = increment(keys=['counter:views'], args=[1])
1214```
1215
1216### Lua Scripts for Complex Operations
1217
1218```python
1219# Rate limiting with Lua
1220rate_limit_script = """
1221local key = KEYS[1]
1222local limit = tonumber(ARGV[1])
1223local window = tonumber(ARGV[2])
1224local current = redis.call('INCR', key)
1225
1226if current == 1 then
1227 redis.call('EXPIRE', key, window)
1228end
1229
1230if current > limit then
1231 return 0
1232else
1233 return 1
1234end
1235"""
1236
1237rate_limiter = r.register_script(rate_limit_script)
1238
1239def is_allowed(user_id: int, limit: int = 100, window: int = 60) -> bool:
1240 """Check if user is within rate limit."""
1241 key = f"rate_limit:{user_id}"
1242 result = rate_limiter(keys=[key], args=[limit, window])
1243 return result == 1
1244
1245# Get-or-set pattern with Lua
1246get_or_set_script = """
1247local value = redis.call('GET', KEYS[1])
1248if value then
1249 return value
1250else
1251 redis.call('SET', KEYS[1], ARGV[1])
1252 redis.call('EXPIRE', KEYS[1], ARGV[2])
1253 return ARGV[1]
1254end
1255"""
1256
1257get_or_set = r.register_script(get_or_set_script)
1258
1259def get_or_compute(key: str, compute_fn, ttl: int = 3600):
1260 """Get value from cache or compute and cache it."""
1261 value = get_or_set(keys=[key], args=["__COMPUTING__", ttl])
1262
1263 if value == "__COMPUTING__":
1264 # We set the placeholder - compute the real value
1265 computed = compute_fn()
1266 r.setex(key, ttl, computed)
1267 return computed
1268
1269 return value
1270```
1271
1272## Production Patterns
1273
1274### High Availability with Sentinel
1275
1276```python
1277from redis.sentinel import Sentinel
1278
1279# Connect to Sentinel
1280sentinel = Sentinel([
1281 ('sentinel1', 26379),
1282 ('sentinel2', 26379),
1283 ('sentinel3', 26379)
1284], socket_timeout=0.5)
1285
1286# Get master connection
1287master = sentinel.master_for('mymaster', socket_timeout=0.5)
1288
1289# Get replica connection (for read-only operations)
1290replica = sentinel.slave_for('mymaster', socket_timeout=0.5)
1291
1292# Use master for writes
1293master.set('key', 'value')
1294
1295# Use replica for reads (optional, for load distribution)
1296value = replica.get('key')
1297```
1298
1299### Async Redis with asyncio
1300
1301```python
1302import asyncio
1303import redis.asyncio as redis
1304
1305async def async_redis_operations():
1306 """Async Redis operations example."""
1307 # Create async connection
1308 r = await redis.from_url("redis://localhost")
1309
1310 try:
1311 # Async operations
1312 await r.set("async_key", "async_value")
1313 value = await r.get("async_key")
1314 print(f"Value: {value}")
1315
1316 # Async pipeline
1317 async with r.pipeline(transaction=True) as pipe:
1318 await pipe.set("key1", "value1")
1319 await pipe.set("key2", "value2")
1320 await pipe.get("key1")
1321 results = await pipe.execute()
1322
1323 print(f"Pipeline results: {results}")
1324
1325 finally:
1326 await r.close()
1327
1328# Run async operations
1329asyncio.run(async_redis_operations())
1330```
1331
1332### Connection Pool Configuration
1333
1334```python
1335# Production-ready connection pool
1336pool = redis.ConnectionPool(
1337 host='localhost',
1338 port=6379,
1339 db=0,
1340 max_connections=50, # Max pool size
1341 socket_timeout=5, # Socket timeout
1342 socket_connect_timeout=5, # Connection timeout
1343 socket_keepalive=True, # Keep TCP connection alive
1344 socket_keepalive_options={
1345 socket.TCP_KEEPIDLE: 60,
1346 socket.TCP_KEEPINTVL: 10,
1347 socket.TCP_KEEPCNT: 3
1348 },
1349 retry_on_timeout=True, # Retry on timeout
1350 health_check_interval=30, # Health check every 30s
1351 decode_responses=True # Auto-decode bytes to strings
1352)
1353
1354r = redis.Redis(connection_pool=pool)
1355```
1356
1357### Error Handling and Resilience
1358
1359```python
1360import redis
1361from redis.exceptions import ConnectionError, TimeoutError
1362import time
1363
1364class ResilientRedisClient:
1365 """Redis client with retry logic and circuit breaker."""
1366
1367 def __init__(self, max_retries: int = 3, backoff: float = 0.1):
1368 self.redis = redis.Redis(
1369 host='localhost',
1370 port=6379,
1371 socket_timeout=5,
1372 retry_on_timeout=True
1373 )
1374 self.max_retries = max_retries
1375 self.backoff = backoff
1376
1377 def get_with_retry(self, key: str, default=None):
1378 """Get value with exponential backoff retry."""
1379 for attempt in range(self.max_retries):
1380 try:
1381 return self.redis.get(key) or default
1382 except (ConnectionError, TimeoutError) as e:
1383 if attempt == self.max_retries - 1:
1384 # Log error and return default
1385 print(f"Redis error after {self.max_retries} attempts: {e}")
1386 return default
1387
1388 # Exponential backoff
1389 wait_time = self.backoff * (2 ** attempt)
1390 time.sleep(wait_time)
1391
1392 def set_with_retry(self, key: str, value: str, ttl: int = None) -> bool:
1393 """Set value with retry logic."""
1394 for attempt in range(self.max_retries):
1395 try:
1396 if ttl:
1397 return self.redis.setex(key, ttl, value)
1398 else:
1399 return self.redis.set(key, value)
1400 except (ConnectionError, TimeoutError) as e:
1401 if attempt == self.max_retries - 1:
1402 print(f"Redis error after {self.max_retries} attempts: {e}")
1403 return False
1404
1405 wait_time = self.backoff * (2 ** attempt)
1406 time.sleep(wait_time)
1407```
1408
1409### Monitoring and Metrics
1410
1411```python
1412def get_redis_info(section: str = None) -> dict:
1413 """Get Redis server information."""
1414 return r.info(section=section)
1415
1416def monitor_memory_usage():
1417 """Monitor Redis memory usage."""
1418 info = r.info('memory')
1419
1420 used_memory = info['used_memory_human']
1421 peak_memory = info['used_memory_peak_human']
1422 memory_fragmentation = info['mem_fragmentation_ratio']
1423
1424 print(f"Used Memory: {used_memory}")
1425 print(f"Peak Memory: {peak_memory}")
1426 print(f"Fragmentation Ratio: {memory_fragmentation}")
1427
1428 return info
1429
1430def monitor_stats():
1431 """Monitor Redis statistics."""
1432 info = r.info('stats')
1433
1434 total_connections = info['total_connections_received']
1435 total_commands = info['total_commands_processed']
1436 ops_per_sec = info['instantaneous_ops_per_sec']
1437
1438 print(f"Total Connections: {total_connections}")
1439 print(f"Total Commands: {total_commands}")
1440 print(f"Ops/sec: {ops_per_sec}")
1441
1442 return info
1443
1444def get_slow_log(count: int = 10):
1445 """Get slow query log."""
1446 slow_log = r.slowlog_get(count)
1447
1448 for entry in slow_log:
1449 print(f"Command: {entry['command']}")
1450 print(f"Duration: {entry['duration']} microseconds")
1451 print(f"Time: {entry['start_time']}")
1452 print("---")
1453
1454 return slow_log
1455```
1456
1457## Best Practices
1458
1459### Key Naming Conventions
1460
1461Use consistent, hierarchical naming:
1462
1463```python
1464# Good naming patterns
1465user:123:profile # User profile data
1466user:123:sessions:abc # User session
1467cache:product:456 # Cached product
1468queue:emails:pending # Email queue
1469lock:resource:789 # Resource lock
1470counter:api:requests:daily # Daily API request counter
1471leaderboard:global:score # Global leaderboard
1472
1473# Avoid
1474u123 # Too cryptic
1475user_profile_123 # Underscores less common
1476123:user # Wrong hierarchy
1477```
1478
1479### Memory Management
1480
1481```python
1482# Set TTL on all temporary data
1483r.setex("temp:data", 3600, value) # Expires in 1 hour
1484
1485# Limit collection sizes
1486r.lpush("activity_log", entry)
1487r.ltrim("activity_log", 0, 999) # Keep only 1000 items
1488
1489# Use appropriate data structures
1490# Hash is more memory-efficient than multiple keys
1491r.hset("user:123", mapping={"name": "Alice", "email": "alice@example.com"})
1492# vs
1493r.set("user:123:name", "Alice")
1494r.set("user:123:email", "alice@example.com")
1495
1496# Monitor memory usage
1497if r.info('memory')['used_memory'] > threshold:
1498 # Implement eviction or cleanup
1499 cleanup_old_data()
1500```
1501
1502### Security
1503
1504```python
1505# Use authentication
1506r = redis.Redis(
1507 host='localhost',
1508 port=6379,
1509 password='your-secure-password',
1510 username='your-username' # Redis 6+
1511)
1512
1513# Use SSL/TLS for production
1514pool = redis.ConnectionPool(
1515 host='redis.example.com',
1516 port=6380,
1517 connection_class=redis.SSLConnection,
1518 ssl_cert_reqs='required',
1519 ssl_ca_certs='/path/to/ca-cert.pem'
1520)
1521
1522# Credential provider pattern
1523from redis import UsernamePasswordCredentialProvider
1524
1525creds_provider = UsernamePasswordCredentialProvider("username", "password")
1526r = redis.Redis(
1527 host="localhost",
1528 port=6379,
1529 credential_provider=creds_provider
1530)
1531```
1532
1533### Testing
1534
1535```python
1536import fakeredis
1537import pytest
1538
1539@pytest.fixture
1540def redis_client():
1541 """Provide fake Redis client for testing."""
1542 return fakeredis.FakeRedis(decode_responses=True)
1543
1544def test_caching(redis_client):
1545 """Test caching logic."""
1546 # Test cache miss
1547 assert redis_client.get("test_key") is None
1548
1549 # Test cache set
1550 redis_client.setex("test_key", 60, "test_value")
1551 assert redis_client.get("test_key") == "test_value"
1552
1553 # Test expiration
1554 assert redis_client.ttl("test_key") <= 60
1555
1556def test_session_management(redis_client):
1557 """Test session operations."""
1558 session_manager = SessionManager(redis_client)
1559
1560 # Create session
1561 session_id = session_manager.create_session(user_id=123)
1562 assert session_id is not None
1563
1564 # Get session
1565 session = session_manager.get_session(session_id)
1566 assert session['user_id'] == 123
1567
1568 # Delete session
1569 assert session_manager.delete_session(session_id) is True
1570 assert session_manager.get_session(session_id) is None
1571```
1572
1573## Examples
1574
1575### Example 1: User Session Management with Redis
1576
1577```python
1578import redis
1579import json
1580import uuid
1581from datetime import datetime, timedelta
1582
1583class UserSessionManager:
1584 """Complete user session management with Redis."""
1585
1586 def __init__(self, redis_client: redis.Redis, ttl: int = 1800):
1587 self.redis = redis_client
1588 self.ttl = ttl
1589
1590 def create_session(self, user_id: int, user_data: dict = None) -> str:
1591 """Create new user session."""
1592 session_id = str(uuid.uuid4())
1593 session_key = f"session:{session_id}"
1594
1595 session_data = {
1596 "user_id": user_id,
1597 "created_at": datetime.utcnow().isoformat(),
1598 "last_accessed": datetime.utcnow().isoformat(),
1599 "data": user_data or {}
1600 }
1601
1602 # Store session with TTL
1603 self.redis.setex(session_key, self.ttl, json.dumps(session_data))
1604
1605 # Track user's active sessions
1606 self.redis.sadd(f"user:{user_id}:sessions", session_id)
1607
1608 return session_id
1609
1610 def get_session(self, session_id: str) -> dict:
1611 """Get session and refresh TTL."""
1612 session_key = f"session:{session_id}"
1613 session_data = self.redis.get(session_key)
1614
1615 if session_data:
1616 session = json.loads(session_data)
1617 session['last_accessed'] = datetime.utcnow().isoformat()
1618
1619 # Refresh TTL
1620 self.redis.setex(session_key, self.ttl, json.dumps(session))
1621
1622 return session
1623
1624 return None
1625
1626 def delete_session(self, session_id: str) -> bool:
1627 """Delete session."""
1628 session = self.get_session(session_id)
1629 if not session:
1630 return False
1631
1632 user_id = session['user_id']
1633
1634 # Remove session
1635 self.redis.delete(f"session:{session_id}")
1636
1637 # Remove from user's session set
1638 self.redis.srem(f"user:{user_id}:sessions", session_id)
1639
1640 return True
1641
1642 def delete_all_user_sessions(self, user_id: int):
1643 """Delete all sessions for a user."""
1644 sessions_key = f"user:{user_id}:sessions"
1645 session_ids = self.redis.smembers(sessions_key)
1646
1647 for session_id in session_ids:
1648 self.redis.delete(f"session:{session_id}")
1649
1650 self.redis.delete(sessions_key)
1651
1652 def get_user_sessions(self, user_id: int) -> list:
1653 """Get all active sessions for a user."""
1654 sessions_key = f"user:{user_id}:sessions"
1655 session_ids = self.redis.smembers(sessions_key)
1656
1657 sessions = []
1658 for session_id in session_ids:
1659 session = self.get_session(session_id)
1660 if session:
1661 session['session_id'] = session_id
1662 sessions.append(session)
1663
1664 return sessions
1665
1666# Usage
1667r = redis.Redis(decode_responses=True)
1668session_mgr = UserSessionManager(r)
1669
1670# Create session
1671session_id = session_mgr.create_session(
1672 user_id=123,
1673 user_data={"role": "admin", "permissions": ["read", "write"]}
1674)
1675
1676# Get session
1677session = session_mgr.get_session(session_id)
1678print(f"User ID: {session['user_id']}")
1679
1680# List all user sessions
1681sessions = session_mgr.get_user_sessions(123)
1682print(f"Active sessions: {len(sessions)}")
1683
1684# Logout (delete session)
1685session_mgr.delete_session(session_id)
1686```
1687
1688### Example 2: Real-Time Leaderboard
1689
1690```python
1691import redis
1692import time
1693
1694class Leaderboard:
1695 """Real-time leaderboard using Redis sorted sets."""
1696
1697 def __init__(self, redis_client: redis.Redis, name: str):
1698 self.redis = redis_client
1699 self.key = f"leaderboard:{name}"
1700
1701 def add_score(self, player_id: str, score: float):
1702 """Add or update player score."""
1703 self.redis.zadd(self.key, {player_id: score})
1704
1705 def increment_score(self, player_id: str, increment: float):
1706 """Increment player score."""
1707 self.redis.zincrby(self.key, increment, player_id)
1708
1709 def get_top(self, count: int = 10) -> list:
1710 """Get top players."""
1711 # ZREVRANGE for highest scores first
1712 players = self.redis.zrevrange(self.key, 0, count - 1, withscores=True)
1713
1714 return [
1715 {
1716 "rank": idx + 1,
1717 "player_id": player_id,
1718 "score": score
1719 }
1720 for idx, (player_id, score) in enumerate(players)
1721 ]
1722
1723 def get_rank(self, player_id: str) -> dict:
1724 """Get player rank and score."""
1725 score = self.redis.zscore(self.key, player_id)
1726 if score is None:
1727 return None
1728
1729 # ZREVRANK for rank (0-indexed, highest first)
1730 rank = self.redis.zrevrank(self.key, player_id)
1731
1732 return {
1733 "player_id": player_id,
1734 "rank": rank + 1 if rank is not None else None,
1735 "score": score
1736 }
1737
1738 def get_around(self, player_id: str, count: int = 5) -> list:
1739 """Get players around a specific player."""
1740 rank = self.redis.zrevrank(self.key, player_id)
1741 if rank is None:
1742 return []
1743
1744 # Get players before and after
1745 start = max(0, rank - count)
1746 end = rank + count
1747
1748 players = self.redis.zrevrange(self.key, start, end, withscores=True)
1749
1750 return [
1751 {
1752 "rank": start + idx + 1,
1753 "player_id": pid,
1754 "score": score,
1755 "is_current": pid == player_id
1756 }
1757 for idx, (pid, score) in enumerate(players)
1758 ]
1759
1760 def get_total_players(self) -> int:
1761 """Get total number of players."""
1762 return self.redis.zcard(self.key)
1763
1764 def remove_player(self, player_id: str) -> bool:
1765 """Remove player from leaderboard."""
1766 return self.redis.zrem(self.key, player_id) > 0
1767
1768# Usage
1769r = redis.Redis(decode_responses=True)
1770leaderboard = Leaderboard(r, "global")
1771
1772# Add scores
1773leaderboard.add_score("alice", 1500)
1774leaderboard.add_score("bob", 2000)
1775leaderboard.add_score("charlie", 1800)
1776leaderboard.increment_score("alice", 200) # alice now at 1700
1777
1778# Get top 10
1779top_players = leaderboard.get_top(10)
1780for player in top_players:
1781 print(f"#{player['rank']}: {player['player_id']} - {player['score']}")
1782
1783# Get player rank
1784alice_stats = leaderboard.get_rank("alice")
1785print(f"Alice is rank {alice_stats['rank']} with {alice_stats['score']} points")
1786
1787# Get players around alice
1788nearby = leaderboard.get_around("alice", count=2)
1789for player in nearby:
1790 marker = " <-- YOU" if player['is_current'] else ""
1791 print(f"#{player['rank']}: {player['player_id']} - {player['score']}{marker}")
1792```
1793
1794### Example 3: Distributed Rate Limiter
1795
1796```python
1797import redis
1798import time
1799
1800class RateLimiter:
1801 """Distributed rate limiter using Redis."""
1802
1803 def __init__(self, redis_client: redis.Redis):
1804 self.redis = redis_client
1805
1806 # Lua script for atomic rate limiting
1807 self.rate_limit_script = self.redis.register_script("""
1808 local key = KEYS[1]
1809 local limit = tonumber(ARGV[1])
1810 local window = tonumber(ARGV[2])
1811
1812 local current = redis.call('INCR', key)
1813
1814 if current == 1 then
1815 redis.call('EXPIRE', key, window)
1816 end
1817
1818 if current > limit then
1819 return {0, limit, current - 1}
1820 else
1821 return {1, limit, current}
1822 end
1823 """)
1824
1825 def check_rate_limit(self, identifier: str, limit: int, window: int) -> dict:
1826 """
1827 Check if request is within rate limit.
1828
1829 Args:
1830 identifier: User ID, IP address, or API key
1831 limit: Maximum requests allowed
1832 window: Time window in seconds
1833
1834 Returns:
1835 dict with allowed (bool), limit, current, remaining
1836 """
1837 key = f"rate_limit:{identifier}:{int(time.time() // window)}"
1838
1839 allowed, max_limit, current = self.rate_limit_script(
1840 keys=[key],
1841 args=[limit, window]
1842 )
1843
1844 return {
1845 "allowed": bool(allowed),
1846 "limit": max_limit,
1847 "current": current,
1848 "remaining": max(0, max_limit - current),
1849 "reset_at": (int(time.time() // window) + 1) * window
1850 }
1851
1852 def sliding_window_check(self, identifier: str, limit: int, window: int) -> dict:
1853 """
1854 Sliding window rate limiter using sorted sets.
1855 More accurate but slightly more expensive.
1856 """
1857 key = f"rate_limit:sliding:{identifier}"
1858 now = time.time()
1859 window_start = now - window
1860
1861 # Remove old entries
1862 self.redis.zremrangebyscore(key, 0, window_start)
1863
1864 # Count current requests
1865 current = self.redis.zcard(key)
1866
1867 if current < limit:
1868 # Add new request
1869 self.redis.zadd(key, {str(now): now})
1870 self.redis.expire(key, window)
1871
1872 return {
1873 "allowed": True,
1874 "limit": limit,
1875 "current": current + 1,
1876 "remaining": limit - current - 1
1877 }
1878 else:
1879 return {
1880 "allowed": False,
1881 "limit": limit,
1882 "current": current,
1883 "remaining": 0
1884 }
1885
1886# Usage
1887r = redis.Redis(decode_responses=True)
1888limiter = RateLimiter(r)
1889
1890# API rate limiting: 100 requests per minute
1891user_id = "user_123"
1892result = limiter.check_rate_limit(user_id, limit=100, window=60)
1893
1894if result["allowed"]:
1895 print(f"Request allowed. {result['remaining']} requests remaining.")
1896 # Process request
1897else:
1898 print(f"Rate limit exceeded. Try again at {result['reset_at']}")
1899 # Return 429 Too Many Requests
1900
1901# More accurate sliding window
1902result = limiter.sliding_window_check(user_id, limit=100, window=60)
1903```
1904
1905### Example 4: Distributed Job Queue
1906
1907```python
1908import redis
1909import json
1910import time
1911import uuid
1912from typing import Optional, Callable
1913
1914class JobQueue:
1915 """Distributed job queue with Redis."""
1916
1917 def __init__(self, redis_client: redis.Redis, queue_name: str = "default"):
1918 self.redis = redis_client
1919 self.queue_name = queue_name
1920 self.queue_key = f"queue:{queue_name}"
1921 self.processing_key = f"queue:{queue_name}:processing"
1922
1923 def enqueue(self, job_type: str, payload: dict, priority: int = 0) -> str:
1924 """
1925 Add job to queue.
1926
1927 Args:
1928 job_type: Type of job (for routing to workers)
1929 payload: Job data
1930 priority: Higher priority = processed first (0 = normal)
1931
1932 Returns:
1933 job_id
1934 """
1935 job_id = str(uuid.uuid4())
1936
1937 job_data = {
1938 "id": job_id,
1939 "type": job_type,
1940 "payload": payload,
1941 "enqueued_at": time.time(),
1942 "attempts": 0
1943 }
1944
1945 # Add to queue (use ZADD for priority queue)
1946 score = -priority # Negative for higher priority first
1947 self.redis.zadd(self.queue_key, {json.dumps(job_data): score})
1948
1949 return job_id
1950
1951 def dequeue(self, timeout: int = 0) -> Optional[dict]:
1952 """
1953 Get next job from queue.
1954
1955 Args:
1956 timeout: Block for this many seconds (0 = no blocking)
1957
1958 Returns:
1959 Job data or None
1960 """
1961 # Get highest priority job (lowest score)
1962 jobs = self.redis.zrange(self.queue_key, 0, 0)
1963
1964 if not jobs:
1965 if timeout > 0:
1966 time.sleep(min(timeout, 1))
1967 return self.dequeue(timeout - 1)
1968 return None
1969
1970 job_json = jobs[0]
1971
1972 # Move to processing set atomically
1973 pipe = self.redis.pipeline()
1974 pipe.zrem(self.queue_key, job_json)
1975 pipe.zadd(self.processing_key, {job_json: time.time()})
1976 pipe.execute()
1977
1978 job_data = json.loads(job_json)
1979 job_data['attempts'] += 1
1980
1981 return job_data
1982
1983 def complete(self, job_data: dict):
1984 """Mark job as completed."""
1985 job_json = json.dumps({
1986 k: v for k, v in job_data.items()
1987 if k != 'attempts'
1988 })
1989
1990 # Remove from processing
1991 self.redis.zrem(self.processing_key, job_json)
1992
1993 def retry(self, job_data: dict, delay: int = 0):
1994 """Retry failed job."""
1995 job_json = json.dumps({
1996 k: v for k, v in job_data.items()
1997 if k != 'attempts'
1998 })
1999
2000 # Remove from processing
2001 self.redis.zrem(self.processing_key, job_json)
2002
2003 # Re-enqueue with delay
2004 if delay > 0:
2005 time.sleep(delay)
2006
2007 self.redis.zadd(self.queue_key, {job_json: 0})
2008
2009 def get_stats(self) -> dict:
2010 """Get queue statistics."""
2011 return {
2012 "queued": self.redis.zcard(self.queue_key),
2013 "processing": self.redis.zcard(self.processing_key)
2014 }
2015
2016# Worker example
2017class Worker:
2018 """Job worker."""
2019
2020 def __init__(self, queue: JobQueue, handlers: dict):
2021 self.queue = queue
2022 self.handlers = handlers
2023
2024 def process_jobs(self):
2025 """Process jobs from queue."""
2026 print("Worker started. Waiting for jobs...")
2027
2028 while True:
2029 job = self.queue.dequeue(timeout=5)
2030
2031 if job:
2032 print(f"Processing job {job['id']} (type: {job['type']})")
2033
2034 try:
2035 # Get handler for job type
2036 handler = self.handlers.get(job['type'])
2037
2038 if handler:
2039 handler(job['payload'])
2040 self.queue.complete(job)
2041 print(f"Job {job['id']} completed")
2042 else:
2043 print(f"No handler for job type: {job['type']}")
2044 self.queue.complete(job)
2045
2046 except Exception as e:
2047 print(f"Job {job['id']} failed: {e}")
2048
2049 if job['attempts'] < 3:
2050 # Retry with exponential backoff
2051 delay = 2 ** job['attempts']
2052 print(f"Retrying in {delay}s...")
2053 self.queue.retry(job, delay=delay)
2054 else:
2055 print(f"Job {job['id']} failed permanently")
2056 self.queue.complete(job)
2057
2058# Usage
2059r = redis.Redis(decode_responses=True)
2060queue = JobQueue(r, "email_queue")
2061
2062# Enqueue jobs
2063job_id = queue.enqueue("send_email", {
2064 "to": "user@example.com",
2065 "subject": "Welcome!",
2066 "body": "Thanks for signing up."
2067}, priority=1)
2068
2069# Define handlers
2070def send_email_handler(payload):
2071 print(f"Sending email to {payload['to']}")
2072 # Email sending logic here
2073 time.sleep(1) # Simulate work
2074
2075handlers = {
2076 "send_email": send_email_handler
2077}
2078
2079# Start worker
2080worker = Worker(queue, handlers)
2081# worker.process_jobs() # This blocks - run in separate process
2082```
2083
2084### Example 5: Real-Time Event Streaming
2085
2086```python
2087import redis
2088import json
2089import time
2090from typing import Callable, Optional
2091
2092class EventStream:
2093 """Real-time event streaming with Redis Streams."""
2094
2095 def __init__(self, redis_client: redis.Redis, stream_name: str):
2096 self.redis = redis_client
2097 self.stream_name = stream_name
2098
2099 def publish(self, event_type: str, data: dict) -> str:
2100 """Publish event to stream."""
2101 event = {
2102 "type": event_type,
2103 "data": json.dumps(data),
2104 "timestamp": time.time()
2105 }
2106
2107 # Add to stream (returns auto-generated ID)
2108 event_id = self.redis.xadd(self.stream_name, event, maxlen=10000)
2109 return event_id
2110
2111 def read_events(self, last_id: str = '0', count: int = 10) -> list:
2112 """Read events from stream."""
2113 events = self.redis.xread(
2114 {self.stream_name: last_id},
2115 count=count,
2116 block=1000 # 1 second timeout
2117 )
2118
2119 if not events:
2120 return []
2121
2122 _, event_list = events[0]
2123
2124 return [
2125 {
2126 "id": event_id,
2127 "type": event_data[b'type'].decode(),
2128 "data": json.loads(event_data[b'data'].decode()),
2129 "timestamp": float(event_data[b'timestamp'])
2130 }
2131 for event_id, event_data in event_list
2132 ]
2133
2134 def create_consumer_group(self, group_name: str):
2135 """Create consumer group for parallel processing."""
2136 try:
2137 self.redis.xgroup_create(
2138 name=self.stream_name,
2139 groupname=group_name,
2140 id='0',
2141 mkstream=True
2142 )
2143 except redis.ResponseError as e:
2144 if "BUSYGROUP" not in str(e):
2145 raise
2146
2147 def consume_events(self, group_name: str, consumer_name: str,
2148 count: int = 10) -> list:
2149 """Consume events as part of consumer group."""
2150 events = self.redis.xreadgroup(
2151 groupname=group_name,
2152 consumername=consumer_name,
2153 streams={self.stream_name: '>'},
2154 count=count,
2155 block=5000
2156 )
2157
2158 if not events:
2159 return []
2160
2161 _, event_list = events[0]
2162
2163 return [
2164 {
2165 "id": event_id,
2166 "type": event_data[b'type'].decode(),
2167 "data": json.loads(event_data[b'data'].decode()),
2168 "timestamp": float(event_data[b'timestamp'])
2169 }
2170 for event_id, event_data in event_list
2171 ]
2172
2173 def acknowledge(self, group_name: str, event_id: str):
2174 """Acknowledge processed event."""
2175 self.redis.xack(self.stream_name, group_name, event_id)
2176
2177 def get_pending(self, group_name: str) -> list:
2178 """Get pending (unacknowledged) events."""
2179 pending = self.redis.xpending_range(
2180 name=self.stream_name,
2181 groupname=group_name,
2182 min='-',
2183 max='+',
2184 count=100
2185 )
2186
2187 return pending
2188
2189# Usage Example: Activity Feed
2190r = redis.Redis()
2191activity_stream = EventStream(r, "user_activity")
2192
2193# Publish events
2194activity_stream.publish("user_signup", {
2195 "user_id": 123,
2196 "email": "alice@example.com"
2197})
2198
2199activity_stream.publish("post_created", {
2200 "user_id": 123,
2201 "post_id": 456,
2202 "title": "My First Post"
2203})
2204
2205# Read events (simple consumer)
2206last_id = '0'
2207while True:
2208 events = activity_stream.read_events(last_id, count=10)
2209
2210 for event in events:
2211 print(f"Event: {event['type']}")
2212 print(f"Data: {event['data']}")
2213 last_id = event['id']
2214
2215 if not events:
2216 break
2217
2218# Consumer group example
2219activity_stream.create_consumer_group("processors")
2220
2221# Worker consuming events
2222while True:
2223 events = activity_stream.consume_events(
2224 group_name="processors",
2225 consumer_name="worker-1",
2226 count=10
2227 )
2228
2229 for event in events:
2230 try:
2231 # Process event
2232 process_event(event)
2233
2234 # Acknowledge
2235 activity_stream.acknowledge("processors", event['id'])
2236 except Exception as e:
2237 print(f"Failed to process event {event['id']}: {e}")
2238 # Event remains unacknowledged for retry
2239```
2240
2241### Example 6: Cache-Aside Pattern with Multi-Level Caching
2242
2243```python
2244import redis
2245import json
2246import hashlib
2247from typing import Optional, Any, Callable
2248
2249class MultiLevelCache:
2250 """Multi-level caching with Redis and local cache."""
2251
2252 def __init__(self, redis_client: redis.Redis,
2253 local_cache_size: int = 100,
2254 local_ttl: int = 60,
2255 redis_ttl: int = 3600):
2256 self.redis = redis_client
2257 self.local_cache = {}
2258 self.local_cache_size = local_cache_size
2259 self.local_ttl = local_ttl
2260 self.redis_ttl = redis_ttl
2261
2262 def _make_key(self, namespace: str, key: str) -> str:
2263 """Generate cache key."""
2264 return f"cache:{namespace}:{key}"
2265
2266 def get(self, namespace: str, key: str,
2267 compute_fn: Optional[Callable] = None) -> Optional[Any]:
2268 """
2269 Get value from cache with fallback to compute function.
2270
2271 Lookup order: Local cache → Redis → Compute function
2272 """
2273 cache_key = self._make_key(namespace, key)
2274
2275 # Level 1: Local cache
2276 if cache_key in self.local_cache:
2277 entry = self.local_cache[cache_key]
2278 if time.time() < entry['expires_at']:
2279 return entry['value']
2280 else:
2281 del self.local_cache[cache_key]
2282
2283 # Level 2: Redis cache
2284 redis_value = self.redis.get(cache_key)
2285 if redis_value:
2286 value = json.loads(redis_value)
2287
2288 # Populate local cache
2289 self._set_local(cache_key, value)
2290
2291 return value
2292
2293 # Level 3: Compute function
2294 if compute_fn:
2295 value = compute_fn()
2296 if value is not None:
2297 self.set(namespace, key, value)
2298 return value
2299
2300 return None
2301
2302 def set(self, namespace: str, key: str, value: Any):
2303 """Set value in both cache levels."""
2304 cache_key = self._make_key(namespace, key)
2305 serialized = json.dumps(value)
2306
2307 # Set in Redis
2308 self.redis.setex(cache_key, self.redis_ttl, serialized)
2309
2310 # Set in local cache
2311 self._set_local(cache_key, value)
2312
2313 def _set_local(self, key: str, value: Any):
2314 """Set value in local cache with LRU eviction."""
2315 # Simple LRU: remove oldest if at capacity
2316 if len(self.local_cache) >= self.local_cache_size:
2317 # Remove oldest entry
2318 oldest_key = min(
2319 self.local_cache.keys(),
2320 key=lambda k: self.local_cache[k]['expires_at']
2321 )
2322 del self.local_cache[oldest_key]
2323
2324 self.local_cache[key] = {
2325 'value': value,
2326 'expires_at': time.time() + self.local_ttl
2327 }
2328
2329 def delete(self, namespace: str, key: str):
2330 """Delete from all cache levels."""
2331 cache_key = self._make_key(namespace, key)
2332
2333 # Delete from Redis
2334 self.redis.delete(cache_key)
2335
2336 # Delete from local cache
2337 if cache_key in self.local_cache:
2338 del self.local_cache[cache_key]
2339
2340 def invalidate_namespace(self, namespace: str):
2341 """Invalidate all keys in namespace."""
2342 pattern = f"cache:{namespace}:*"
2343
2344 # Delete from Redis
2345 for key in self.redis.scan_iter(match=pattern, count=100):
2346 self.redis.delete(key)
2347
2348 # Delete from local cache
2349 to_delete = [
2350 k for k in self.local_cache.keys()
2351 if k.startswith(f"cache:{namespace}:")
2352 ]
2353 for k in to_delete:
2354 del self.local_cache[k]
2355
2356# Usage
2357r = redis.Redis(decode_responses=True)
2358cache = MultiLevelCache(r)
2359
2360def get_user(user_id: int) -> dict:
2361 """Get user with multi-level caching."""
2362 return cache.get(
2363 namespace="users",
2364 key=str(user_id),
2365 compute_fn=lambda: database.query_user(user_id)
2366 )
2367
2368# First call: Queries database, caches result
2369user = get_user(123)
2370
2371# Second call: Returns from local cache (fastest)
2372user = get_user(123)
2373
2374# Update user
2375def update_user(user_id: int, data: dict):
2376 database.update_user(user_id, data)
2377
2378 # Invalidate cache
2379 cache.delete("users", str(user_id))
2380
2381# Invalidate all user caches
2382cache.invalidate_namespace("users")
2383```
2384
2385### Example 7: Geo-Location with Redis
2386
2387```python
2388import redis
2389
2390class GeoLocation:
2391 """Geo-spatial indexing and queries with Redis."""
2392
2393 def __init__(self, redis_client: redis.Redis, index_name: str):
2394 self.redis = redis_client
2395 self.key = f"geo:{index_name}"
2396
2397 def add_location(self, location_id: str, longitude: float, latitude: float):
2398 """Add location to geo index."""
2399 self.redis.geoadd(self.key, longitude, latitude, location_id)
2400
2401 def add_locations(self, locations: list):
2402 """Batch add locations.
2403
2404 Args:
2405 locations: List of (location_id, longitude, latitude) tuples
2406 """
2407 self.redis.geoadd(self.key, *[
2408 item for loc in locations
2409 for item in (loc[1], loc[2], loc[0])
2410 ])
2411
2412 def get_position(self, location_id: str) -> tuple:
2413 """Get coordinates of a location."""
2414 result = self.redis.geopos(self.key, location_id)
2415 if result and result[0]:
2416 return result[0] # (longitude, latitude)
2417 return None
2418
2419 def find_nearby(self, longitude: float, latitude: float,
2420 radius: float, unit: str = 'km', count: int = None) -> list:
2421 """
2422 Find locations within radius.
2423
2424 Args:
2425 longitude: Center longitude
2426 latitude: Center latitude
2427 radius: Search radius
2428 unit: Distance unit ('m', 'km', 'mi', 'ft')
2429 count: Maximum results
2430 """
2431 args = {
2432 'longitude': longitude,
2433 'latitude': latitude,
2434 'radius': radius,
2435 'unit': unit,
2436 'withdist': True,
2437 'withcoord': True,
2438 'sort': 'ASC'
2439 }
2440
2441 if count:
2442 args['count'] = count
2443
2444 results = self.redis.georadius(self.key, **args)
2445
2446 return [
2447 {
2448 'location_id': location_id,
2449 'distance': distance,
2450 'coordinates': (longitude, latitude)
2451 }
2452 for location_id, distance, (longitude, latitude) in results
2453 ]
2454
2455 def find_nearby_member(self, location_id: str, radius: float,
2456 unit: str = 'km', count: int = None) -> list:
2457 """Find locations near an existing member."""
2458 args = {
2459 'member': location_id,
2460 'radius': radius,
2461 'unit': unit,
2462 'withdist': True,
2463 'sort': 'ASC'
2464 }
2465
2466 if count:
2467 args['count'] = count
2468
2469 results = self.redis.georadiusbymember(self.key, **args)
2470
2471 return [
2472 {
2473 'location_id': loc_id,
2474 'distance': distance
2475 }
2476 for loc_id, distance in results
2477 if loc_id != location_id # Exclude self
2478 ]
2479
2480 def distance_between(self, location_id1: str, location_id2: str,
2481 unit: str = 'km') -> float:
2482 """Calculate distance between two locations."""
2483 return self.redis.geodist(self.key, location_id1, location_id2, unit)
2484
2485# Usage Example: Restaurant finder
2486r = redis.Redis(decode_responses=True)
2487restaurants = GeoLocation(r, "restaurants")
2488
2489# Add restaurants
2490restaurants.add_locations([
2491 ("rest1", -122.4194, 37.7749), # San Francisco
2492 ("rest2", -122.4068, 37.7849),
2493 ("rest3", -122.4312, 37.7652),
2494])
2495
2496# Find restaurants near coordinates
2497nearby = restaurants.find_nearby(
2498 longitude=-122.4194,
2499 latitude=37.7749,
2500 radius=5,
2501 unit='km',
2502 count=10
2503)
2504
2505for restaurant in nearby:
2506 print(f"{restaurant['location_id']}: {restaurant['distance']:.2f} km away")
2507
2508# Find restaurants near a specific restaurant
2509similar = restaurants.find_nearby_member("rest1", radius=2, unit='km')
2510
2511# Get distance between two restaurants
2512distance = restaurants.distance_between("rest1", "rest2", unit='km')
2513print(f"Distance: {distance:.2f} km")
2514```
2515
2516## Quick Reference
2517
2518### Common Operations
2519
2520```python
2521# Connection
2522r = redis.Redis(host='localhost', port=6379, db=0, decode_responses=True)
2523
2524# Strings
2525r.set('key', 'value')
2526r.setex('key', 3600, 'value') # With TTL
2527r.get('key')
2528r.incr('counter')
2529
2530# Hashes
2531r.hset('user:123', 'name', 'Alice')
2532r.hset('user:123', mapping={'name': 'Alice', 'age': 30})
2533r.hget('user:123', 'name')
2534r.hgetall('user:123')
2535
2536# Lists
2537r.lpush('queue', 'item')
2538r.rpush('queue', 'item')
2539r.lpop('queue')
2540r.lrange('queue', 0, -1)
2541
2542# Sets
2543r.sadd('tags', 'python', 'redis')
2544r.smembers('tags')
2545r.sismember('tags', 'python')
2546
2547# Sorted Sets
2548r.zadd('leaderboard', {'alice': 100, 'bob': 200})
2549r.zrange('leaderboard', 0, -1, withscores=True)
2550r.zrank('leaderboard', 'alice')
2551
2552# Expiration
2553r.expire('key', 3600)
2554r.ttl('key')
2555
2556# Pipelining
2557pipe = r.pipeline()
2558pipe.set('key1', 'value1')
2559pipe.set('key2', 'value2')
2560results = pipe.execute()
2561```
2562
2563### Time Complexity
2564
2565- GET, SET: O(1)
2566- HGET, HSET: O(1)
2567- LPUSH, RPUSH, LPOP, RPOP: O(1)
2568- SADD, SREM, SISMEMBER: O(1)
2569- ZADD, ZREM: O(log(N))
2570- ZRANGE, ZREVRANGE: O(log(N)+M) where M is result size
2571- SCAN, SSCAN, HSCAN, ZSCAN: O(1) per iteration
2572
2573---
2574
2575**Skill Version**: 1.0.0
2576**Last Updated**: October 2025
2577**Skill Category**: State Management, Distributed Systems, Performance Optimization
2578**Compatible With**: redis-py, Redis 6.0+, Redis 7.0+