# Resilience Patterns

> Apply reliability primitives — capped exponential backoff with jitter, circuit breakers, timeouts, and idempotency keys — to every outbound call and mutating endpoint. Use when integrating an external service, when retries cause duplicate effects, or before shipping a payment/order flow. Not for job-runner retry config specifically (use background-jobs) or webhook-delivery specifics (use webhook-design, which reuses these primitives).

- Skill: `jaykim88/resilience-patterns` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jaykim88/resilience-patterns`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jaykim88/resilience-patterns/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- License: MIT
- Author: JayKim88 (https://skillmd.com/u/jaykim88)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/jaykim88/resilience-patterns

---


# Resilience Patterns

## Purpose
Make every outbound call and every mutating endpoint survive partial failure without causing duplicate effects or cascading outages. Retries, timeouts, breakers, and idempotency keys are the primitives — apply them deliberately, not reflexively.

**Universal** — capped exponential backoff + jitter, circuit breakers, timeout-from-p99, and idempotency keys are distributed-systems primitives independent of language; only the library differs.

## Procedure

1. **Set timeouts from p99.9 latency — never infinite**
   - Measure the dependency's p99.9; set timeout slightly above it
   - No timeout = one slow dependency exhausts your connection pool → cascading failure

2. **Retry ONLY idempotent operations**
   - GET / PUT / DELETE are naturally idempotent → safe to retry
   - POST is NOT → retry only with an idempotency key (step 4)
   - Never blindly retry a non-idempotent mutation

3. **Use capped exponential backoff + full jitter**
   - `delay = random(0, min(cap, base * 2^attempt))`
   - Full jitter prevents thundering herd (synchronized retries hammering a recovering service)
   - Cap total attempts (e.g., 3-5); after cap, fail to a fallback or DLQ

4. **Idempotency keys for mutations**
   - Client sends `Idempotency-Key: <uuid>` header on POST
   - Server stores key → result mapping; on replay, return the stored result (don't re-execute)
   - Store the key in the SAME transaction as the business effect

5. **Circuit breaker for repeatedly-failing dependencies**
   - Open the breaker after N consecutive failures → fail fast instead of waiting on timeouts
   - Half-open after a cooldown to test recovery
   - Pair with a fallback (cached value, degraded response, queue-for-later)

5b. **Bulkhead: isolate critical paths from one bad dependency**
   - Partition resources (connection pools, worker concurrency) per dependency so one slow integration can't drain the pool that serves the rest
   - Without bulkheading, your healthy endpoints fail because the unhealthy dependency holds every connection

5c. **Define a fallback chain for high-availability paths**
   - Primary → secondary → cached/degraded → user-friendly error
   - Each rung must be decided in advance (not improvised mid-incident); annotate which rung the response came from so observability sees the degradation

5d. **Propagate deadlines across hops**
   - When request A times out at 800ms, its outbound call to B shouldn't have a 2s timeout — by the time B returns, A is already done. Pass the remaining budget (header or context) so each hop's timeout = min(local-budget, deadline-remaining)
   - Otherwise downstream services do work nobody waits for

6. **Validate (validation loop)**
   - Simulate dependency failure (kill the service / inject latency)
   - Verify: no duplicate effects on retry, no cascading timeout, breaker opens and recovers
   - If duplicates occur → idempotency key not applied correctly; fix and re-test

## Anti-patterns

| ❌ Anti-pattern | ✅ Correct |
|---|---|
| Retry a POST with no idempotency key | Idempotency-Key header + stored result |
| Exponential backoff without jitter | Capped exponential + full jitter |
| No timeout on outbound call | Timeout from dependency p99.9 |
| Infinite retries | Cap attempts → fallback / DLQ |
| Catch-all retry on every error | Retry only transient errors (timeout/503), not 4xx |
| One bad dependency drains the shared connection pool | Bulkhead per dependency (separate pool / concurrency limit) |
| Improvising the fallback during the incident | Predeclared fallback chain (primary → secondary → degraded) |
| 2s downstream timeout under a 800ms caller deadline | Propagate the deadline; downstream timeout = min(local, remaining) |

## Severity tiers

| Tier | Examples | Action SLA |
|---|---|---|
| **Critical** | Payment/order POST retried without idempotency key (duplicate charges); no timeout on a synchronous external call | Block release; fix immediately |
| **Major** | Backoff without jitter; missing circuit breaker on a known-flaky dependency | Fix this sprint |
| **Minor** | Retry cap slightly high; breaker cooldown untuned | Schedule within 2 sprints |

## Completion Criteria
- [ ] Every outbound call has a timeout derived from p99.9
- [ ] Every POST that can be retried uses an idempotency key
- [ ] Backoff is capped + jittered
- [ ] Flaky dependencies wrapped in a circuit breaker + fallback
- [ ] Failure simulation confirms no duplicate effects

## Output
- **Resilience config**: timeout/retry/breaker policy per dependency (documented)
- **Idempotency table**: schema + the transaction wiring
- **Commit format**: `feat(resilience): add idempotency key to <endpoint>` / `fix(resilience): cap backoff + jitter on <client>`

## Implementation

### TypeScript + NestJS (default)
- Library: **Cockatiel** (composable retry + circuit breaker + timeout + bulkhead)
- Idempotency: store `idempotency_key → response` in Postgres, checked in the same Prisma `$transaction` as the write
- HTTP client: wrap `fetch`/`axios` calls in a Cockatiel policy

### Other stacks
- **Python / FastAPI**: `tenacity` (retry/backoff) + `pybreaker` (circuit breaker)
- **Go**: `cenkalti/backoff` + `sony/gobreaker`
- **Universal**: idempotency key pattern is HTTP-level (Stripe's `Idempotency-Key` is the reference); jitter formula is math, not library-specific

## Related skills
- `webhook-design` — webhook delivery uses the same retry + idempotency primitives
- `background-jobs` — job retries must be idempotent
- `transaction-management` — idempotency keys often stored in the same transaction as the effect

## Reference
- **Key insight encoded**: Cap exponential backoff and add full jitter to prevent thundering herd; retry only idempotent operations (use a Stripe-style `Idempotency-Key` for POST). Set timeouts from p99.9, not guesses.