Resilience and Reliability Patterns

Resilience and Reliability Patterns

Essential patterns for building fault-tolerant distributed systems.

Resilience Patterns

Circuit Breaker

Purpose: Prevent cascading failures by failing fast when a dependency is unhealthy.

How It Works:

States:
1. CLOSED (normal operation)
   - Requests pass through
   - Track failure rate
   - If failures exceed threshold → OPEN

2. OPEN (failing fast)
   - Immediately reject requests
   - Return fallback response
   - After timeout period → HALF_OPEN

3. HALF_OPEN (testing recovery)
   - Allow limited test requests
   - If successful → CLOSED
   - If failed → OPEN

Configuration:
- Failure threshold: 50% failures in 10 requests
- Timeout: 30 seconds in OPEN state
- Success threshold: 2 consecutive successes in HALF_OPEN

Implementation Example:

# Using resilience4j-like pattern
@CircuitBreaker(
    name="payment-service",
    fallbackMethod="paymentFallback",
    failureThreshold=50,
    waitDurationInOpenState=30000,  # 30s
    permittedNumberOfCallsInHalfOpenState=3
)
async def process_payment(order_id: str, amount: float):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{PAYMENT_SERVICE_URL}/payments",
            json={"orderId": order_id, "amount": amount},
            timeout=5.0
        )
        return response.json()

async def paymentFallback(order_id: str, amount: float, exception):
    # Log the failure
    logger.error(f"Payment service unavailable: {exception}")
    # Return graceful degradation
    return {
        "status": "pending",
        "message": "Payment processing delayed, will retry"
    }

When to Use:

Apply circuit breakers to:
✓ External service calls
✓ Database queries
✓ Third-party APIs
✓ Microservice-to-microservice calls

Configuration Guidelines:
- Fast services (p99 < 100ms): 5s timeout, 10s circuit open
- Medium services (p99 < 1s): 10s timeout, 30s circuit open
- Slow services (p99 > 1s): 30s timeout, 60s circuit open

Retry Pattern

Purpose: Handle transient failures by retrying operations.

Strategies:

1. Exponential Backoff:

Retry delays: 100ms, 200ms, 400ms, 800ms, 1600ms

Benefits:
- Reduces load during incidents
- Gives service time to recover
- Prevents thundering herd

Implementation:
attempts = 0
max_attempts = 5
base_delay = 0.1  # 100ms

while attempts < max_attempts:
    try:
        return await make_request()
    except TransientError as e:
        attempts += 1
        if attempts == max_attempts:
            raise
        delay = base_delay * (2 ** attempts) + random.uniform(0, 0.1)
        await asyncio.sleep(delay)

2. Retry with Jitter:

Why: Prevents synchronized retries (thundering herd)

Full Jitter:
delay = random.uniform(0, base_delay * (2 ** attempt))

Decorrelated Jitter:
delay = min(cap, random.uniform(base, previous_delay * 3))

Recommended: Decorrelated jitter for production systems

3. Idempotency Keys:

Problem: Retries can cause duplicate operations

Solution: Idempotency keys
POST /api/v1/payments
Headers:
  Idempotency-Key: uuid-12345

Server Logic:
1. Check if operation with this key already processed
2. If yes, return cached response
3. If no, process and cache result
4. Cache for 24 hours

Ensures safe retries even for non-idempotent operations

Retry Best Practices:

DO:
✓ Only retry transient errors (timeout, 503, 429)
✓ Use exponential backoff with jitter
✓ Set maximum retry attempts (3-5)
✓ Implement overall timeout
✓ Use idempotency keys for writes
✓ Log each retry attempt

DON'T:
✗ Retry client errors (400, 401, 404)
✗ Retry without backoff (causes load spikes)
✗ Infinite retries
✗ Retry non-idempotent operations without safeguards

Bulkhead Pattern

Purpose: Isolate resources to prevent total system failure.

Thread Pool Isolation:

Concept: Separate thread pools for different operations

Example:
- Payment Service Thread Pool: 20 threads
- Inventory Service Thread Pool: 20 threads
- Notification Service Thread Pool: 10 threads

If payment service becomes slow:
- Only payment thread pool exhausted
- Inventory and notification still work
- System partially degraded, not completely down

Connection Pool Isolation:

Database Connection Pools:
- Read-only queries: 50 connections
- Write queries: 20 connections
- Reporting queries: 10 connections

Heavy reporting query won't starve transactional operations

Rate Limiting per Tenant:

Multi-tenant SaaS application:

tenant-a: 1000 requests/minute
tenant-b: 1000 requests/minute
tenant-c: 1000 requests/minute

If tenant-a floods the system:
- Only tenant-a throttled
- tenant-b and tenant-c unaffected

Implementation:

# Using semaphores for concurrency limits
class BulkheadExecutor:
    def __init__(self):
        self.payment_semaphore = asyncio.Semaphore(20)
        self.inventory_semaphore = asyncio.Semaphore(20)
        self.notification_semaphore = asyncio.Semaphore(10)

    async def call_payment_service(self, data):
        async with self.payment_semaphore:
            return await payment_service.call(data)

    async def call_inventory_service(self, data):
        async with self.inventory_semaphore:
            return await inventory_service.call(data)

Timeout Pattern

Purpose: Prevent indefinite waiting for responses.

Timeout Types:

1. Connection Timeout:

Time allowed to establish connection

Recommended: 2-5 seconds
If takes longer, network likely has issues

httpx.AsyncClient(timeout=httpx.Timeout(connect=3.0))

2. Read Timeout:

Time allowed to receive response after connection

Varies by service:
- Fast APIs: 5 seconds
- Database queries: 10 seconds
- Complex processing: 30 seconds

httpx.AsyncClient(timeout=httpx.Timeout(read=10.0))

3. Total Timeout:

Overall time budget for entire operation

Example: User checkout flow
- Total budget: 30 seconds
- Payment service: 10 seconds
- Inventory check: 5 seconds
- Order creation: 5 seconds
- Buffer: 10 seconds

async with asyncio.timeout(30):
    result = await complete_checkout()

Timeout Best Practices:

Timeouts Hierarchy:
Parent timeout > sum of child timeouts

Request → API Gateway (30s timeout)
  → Service A (10s timeout)
    → Service B (5s timeout)
      → Database (2s timeout)

Set timeouts everywhere:
✓ HTTP clients
✓ Database connections
✓ Message consumers
✓ gRPC calls
✓ Cache operations

Distributed Transaction Patterns

Saga Pattern

Purpose: Manage distributed transactions across services.

Choreography-Based Saga:

Example: Order Creation Saga

Events:
1. OrderService: order.created
2. PaymentService: payment.completed OR payment.failed
3. InventoryService: inventory.reserved OR inventory.reservation.failed
4. ShippingService: shipment.created

Compensating Transactions:
If inventory.reservation.failed:
  → PaymentService listens → refund.initiated
  → OrderService listens → order.cancelled

Pros:
- Decentralized
- No single point of failure
- Services autonomous

Cons:
- Difficult to track saga state
- Complex debugging
- No saga-wide timeout

Orchestration-Based Saga:

Example: Order Saga Orchestrator

Saga Steps:
1. Create Order (OrderService)
2. Charge Payment (PaymentService)
3. Reserve Inventory (InventoryService)
4. Create Shipment (ShippingService)

Orchestrator Logic:
step1_result = await order_service.create_order()
if not step1_result.success:
    return failure("Order creation failed")

step2_result = await payment_service.charge(amount)
if not step2_result.success:
    await order_service.cancel_order(step1_result.order_id)
    return failure("Payment failed")

step3_result = await inventory_service.reserve(items)
if not step3_result.success:
    await payment_service.refund(step2_result.payment_id)
    await order_service.cancel_order(step1_result.order_id)
    return failure("Inventory unavailable")

# Continue saga...

Pros:
- Clear workflow
- Centralized monitoring
- Easy to understand

Cons:
- Orchestrator complexity
- Potential bottleneck
- Coupling to orchestrator

Saga State Management:

Persist saga state to handle failures:

CREATE TABLE saga_instances (
    saga_id UUID PRIMARY KEY,
    saga_type VARCHAR(50),
    current_step VARCHAR(50),
    status VARCHAR(20),
    payload JSONB,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

On orchestrator restart:
- Load incomplete sagas
- Resume from last completed step
- Execute remaining steps or compensations

Event Sourcing

Purpose: Store all state changes as events, derive current state by replaying.

Implementation:

Traditional Approach:
UPDATE orders SET status = 'shipped' WHERE id = 123;
(Lost: when shipped, by whom, from where)

Event Sourcing Approach:
Events:
1. OrderPlaced { orderId, customerId, items, timestamp }
2. PaymentReceived { orderId, amount, paymentId, timestamp }
3. OrderShipped { orderId, trackingNumber, carrier, timestamp }

Current state = replay all events

Event Store:

CREATE TABLE events (
    event_id UUID PRIMARY KEY,
    aggregate_id UUID,
    aggregate_type VARCHAR(50),
    event_type VARCHAR(100),
    event_data JSONB,
    version INTEGER,
    timestamp TIMESTAMP,
    correlation_id UUID
);

CREATE INDEX idx_aggregate ON events(aggregate_id, version);

Guarantees:
- Events immutable
- Events ordered by version
- Optimistic locking prevents conflicts

Benefits:

✓ Full audit trail
✓ Time travel (replay to any point)
✓ Event replay for debugging
✓ Multiple read models from same events
✓ Temporal queries ("show orders as of yesterday")

Challenges:
✗ Eventual consistency
✗ Event schema evolution
✗ Snapshot strategy needed
✗ Increased storage

CQRS (Command Query Responsibility Segregation)

Purpose: Separate read and write models for different optimization strategies.

Architecture:

Write Side (Command):
- Receives commands (CreateOrder, UpdateInventory)
- Validates business rules
- Stores events in event store
- Optimized for consistency and writes

Read Side (Query):
- Listens to events
- Updates denormalized read models
- Optimized for queries
- Eventual consistency

Example:
Command: CreateOrder
  → Order aggregate validates
  → Publishes OrderCreated event
  → Event stored in event store

Query Side:
  → Listens to OrderCreated
  → Updates order_summary table (denormalized)
  → Updates customer_order_history (different view)
  → Updates order_analytics (aggregated metrics)

Read Models:

Multiple specialized views from same events:

1. Order Detail View (for customer):
   { orderId, items, status, total, estimatedDelivery }

2. Order List View (for admin):
   { orderId, customerName, orderDate, status, total }

3. Analytics View:
   { date, totalOrders, totalRevenue, averageOrderValue }

Each optimized for specific query patterns

Fault Tolerance Patterns

Health Checks

Types:

1. Liveness Probe:

Purpose: Is the service alive?

Endpoint: GET /health/live

Returns 200 if:
- Application process running
- Not deadlocked

Kubernetes Action:
- If fails: Restart container

2. Readiness Probe:

Purpose: Is the service ready to receive traffic?

Endpoint: GET /health/ready

Returns 200 if:
- Database connection pool healthy
- Cache accessible
- Downstream dependencies responsive

Kubernetes Action:
- If fails: Remove from load balancer
- Don't send traffic until ready

3. Startup Probe:

Purpose: Has the service finished initialization?

Endpoint: GET /health/startup

For slow-starting applications:
- Prevents premature liveness checks
- Allows longer startup time

Implementation:

@app.get("/health/live")
async def liveness():
    return {"status": "alive"}

@app.get("/health/ready")
async def readiness():
    checks = {
        "database": await check_database(),
        "cache": await check_cache(),
        "payment_service": await check_payment_service()
    }

    all_healthy = all(checks.values())
    status_code = 200 if all_healthy else 503

    return JSONResponse(
        status_code=status_code,
        content={"status": "ready" if all_healthy else "not ready", "checks": checks}
    )

Graceful Degradation

Purpose: Provide reduced functionality when dependencies fail.

Strategies:

1. Cached Responses:

async def get_product_recommendations(user_id):
    try:
        async with circuit_breaker:
            return await ml_service.get_recommendations(user_id)
    except ServiceUnavailable:
        # Fallback to cached popular products
        return await cache.get_popular_products()

2. Default Values:

async def get_user_preferences(user_id):
    try:
        return await preferences_service.get(user_id)
    except ServiceUnavailable:
        # Return sensible defaults
        return {
            "language": "en",
            "currency": "USD",
            "theme": "light"
        }

3. Feature Toggles:

if feature_flags.is_enabled("personalized_recommendations"):
    recommendations = await ml_service.get_recommendations()
else:
    # Fallback to simple algorithm
    recommendations = await get_popular_products()

Summary

Resilience patterns are mandatory in distributed systems. Layer multiple patterns for defense in depth:

Essential Stack:

1. Timeouts (prevent hanging)
2. Retries with backoff (handle transient errors)
3. Circuit breakers (prevent cascading failures)
4. Bulkheads (isolate failures)
5. Health checks (enable auto-healing)
6. Graceful degradation (maintain partial functionality)

Choose Saga Pattern When:

• Distributed transaction needed
• Strong consistency not required
• Compensating transactions possible

Choose Event Sourcing When:

• Full audit trail required
• Temporal queries needed
• Multiple read models beneficial

Always test failure scenarios. Use chaos engineering to validate resilience.