Resilience Patterns

When to Use

• Use when: a slow downstream service causes your service to hang
• Use when: retry storms are making outages worse
• Use when: one noisy tenant is taking down shared resources
• Use when: designing a new service that calls external APIs
• Do NOT use for: database query optimization — use database-patterns

The Four Core Patterns

1. Timeout

Every outbound call must have a deadline. Without it, one slow dependency hangs all threads.

Connection timeout: 1–3s  (time to establish connection)
Read timeout:       5–30s (time to wait for response — match the operation's SLO)
Total timeout:      always less than your own SLO to leave time for retry

Rules:

• Set timeout at call site, not just client config — context-specific timeouts
• Timeout ≠ error; have a fallback for timeout case
• Cascade: if your caller has a 10s SLO, your downstream timeout must be < 8s

2. Retry with Exponential Backoff + Jitter

attempt 1: immediate
attempt 2: wait 1s   + jitter(0–500ms)
attempt 3: wait 2s   + jitter(0–500ms)
attempt 4: wait 4s   + jitter(0–500ms)
max attempts: 3–5 (beyond that, the service is probably down — stop)

Only retry idempotent operations. Never retry: payment charge, send email, create order (unless idempotency key is used).

Jitter is mandatory — without it, all retrying clients synchronize and create a retry storm.

wait = min(base * (2 ** attempt), max_wait) + random.uniform(0, jitter_cap)

3. Circuit Breaker

CLOSED (normal)  →  failure rate > threshold  →  OPEN (reject calls)
OPEN             →  after timeout (30–60s)     →  HALF-OPEN (probe one call)
HALF-OPEN        →  probe succeeds             →  CLOSED
HALF-OPEN        →  probe fails                →  OPEN

Configuration:

failure_threshold:    50% of calls in 10s window → OPEN
success_threshold:    3 consecutive successes in HALF-OPEN → CLOSED
open_duration:        30s before probing
minimum_calls:        10 calls before evaluating failure rate

In OPEN state: return cached response, fallback, or explicit error — never queue indefinitely.

4. Bulkhead

Isolate resources so one slow consumer can't exhaust shared capacity.

Without bulkhead:
  [All callers] → [shared 100-thread pool] → [downstream]
  If downstream slows: all 100 threads fill up → service dies

With bulkhead:
  [Payment calls] → [10-thread pool for payments]
  [Search calls]  → [20-thread pool for search]
  [Reports calls] → [5-thread pool for reports]
  Payments slow → only payment pool fills; search still works

Fallback Strategies

When resilience pattern triggers, always have a fallback:

Rate Limiting

Protect your service from callers, not just from dependencies.

Algorithms

Headers to return (standard)

X-RateLimit-Limit:     100
X-RateLimit-Remaining: 42
X-RateLimit-Reset:     1716300000   (Unix epoch when limit resets)
Retry-After:           60           (seconds to wait — on 429 response)

Pattern Selection Guide

Anti-Fake-Pass Rules

Before claiming a service is resilient, you MUST show:

• Every outbound call has an explicit timeout (not default infinite)
• Retries: exponential backoff + jitter, only on idempotent operations
• Circuit breaker configured with failure threshold and open duration
• Fallback defined for each resilience trigger — not just "return 500"
• Bulkheads: separate thread/connection pools for critical vs non-critical dependencies

Reference: gates/anti-fake-pass-gate.md