Distributed Systems

Distributed Systems Patterns

Comprehensive patterns for building reliable distributed systems. Each category has individual rule files in rules/ loaded on-demand.

Quick Reference

Total: 11 rules across 6 categories. Removed topics point at first-party sources in Upstream coverage; ork-specific scars live in references/ork-delta.md.

Quick Start

# Redis distributed lock with Lua scripts
async with RedisLock(redis_client, "payment:order-123"):
    await process_payment(order_id)

# Circuit breaker for external APIs
@circuit_breaker(failure_threshold=5, recovery_timeout=30)
@retry(max_attempts=3, base_delay=1.0)
async def call_external_api():
    ...

# Idempotent API endpoint
@router.post("/payments")
async def create_payment(
    data: PaymentCreate,
    idempotency_key: str = Header(..., alias="Idempotency-Key"),
):
    return await idempotent_execute(db, idempotency_key, "/payments", process)

# Token bucket rate limiting
limiter = TokenBucketLimiter(redis_client, capacity=100, refill_rate=10)
if await limiter.is_allowed(f"user:{user_id}"):
    await handle_request()

Distributed Locks

Coordinate exclusive access to resources across multiple service instances.

Redis single-node locks, Redlock quorum, and PostgreSQL advisory locks are first-party documented; see Upstream coverage.

Resilience

Production-grade fault tolerance for distributed systems.

Idempotency

Ensure operations can be safely retried without unintended side effects.

Event-consumer dedup and database-backed idempotency storage follow the Stripe pattern; see Upstream coverage.

Rate Limiting

Protect APIs with distributed rate limiting using Redis.

SlowAPI + Redis wiring and tiered limits are first-party documented; see Upstream coverage.

Edge Computing

Edge runtime patterns for Cloudflare Workers, Vercel Edge, and Deno Deploy.

Event-Driven

Event sourcing, CQRS, saga orchestration, and reliable messaging patterns.

Upstream coverage (do not restate)

These topics were removed from this skill on 2026-07-31 (wrap-plus-delta thinning) because a first-party source maintains them. Consult the source; do not re-add tutorials here. Ork-specific scars for these topics live in references/ork-delta.md.

Key Decisions

When NOT to Use

No separate event-sourcing/saga/CQRS skills exist; they are rules within distributed-systems. But most projects never need them.

Rule of thumb: If you have a single server process, you do not need distributed systems patterns. Use in-process alternatives. Add distribution only when you actually have multiple instances.

Anti-Patterns (FORBIDDEN)

# LOCKS: Never forget TTL (causes deadlocks)
await redis.set(f"lock:{name}", "1")  # WRONG - no expiry!

# LOCKS: Never release without owner check
await redis.delete(f"lock:{name}")  # WRONG - might release others' lock

# RESILIENCE: Never retry non-retryable errors
@retry(max_attempts=5, retryable_exceptions={Exception})  # Retries 401!

# RESILIENCE: Never put retry outside circuit breaker
@retry  # Would retry when circuit is open!
@circuit_breaker
async def call(): ...

# IDEMPOTENCY: Never use non-deterministic keys
key = str(uuid.uuid4())  # Different every time!

# IDEMPOTENCY: Never cache error responses
if response.status_code >= 400:
    await cache_response(key, response)  # Errors should retry!

# RATE LIMITING: Never use in-memory counters in distributed systems
request_counts = {}  # Lost on restart, not shared across instances

Detailed Documentation

Related Skills

• caching - Redis caching patterns, cache as fallback
• background-jobs - Job deduplication, async processing with retry
• observability-monitoring - Metrics and alerting for circuit breaker state changes
• error-handling-rfc9457 - Structured error responses for resilience failures
• auth-patterns - API key management, authentication integration