rockclaver Resilience Patterns
Pair with idempotent-financial-workflows (retries) and observability-instrumentation (visibility).
Workflow
- Map dependencies: for each outbound call, note who's called (money-moving/read-only/best-effort), latency budget, and retry-safety.
- Apply the right pattern:
- Timeout: explicit, short connect/read timeout on every call — no unbounded waits.
- Retry: only transient, idempotent ops; backoff+jitter, capped attempts; never retry a non-idempotent move without an idempotency key.
- Circuit breaker: trips after N failures so a dead provider fails fast; half-open probes recovery.
- Bulkhead: isolate pools/concurrency per dependency so one slow provider can't starve others.
- Fallback: serve cached/stale data, queue for later, or return "unavailable" instead of hanging.
- Backpressure: shed or queue load when downstream is saturated instead of piling on.
- Decide degradation: best-effort calls fail open; critical money moves fail closed (stop, persist intent, surface it, retry later).
- Test: timeout/slow response; 5xx/429 recovery; breaker open→half-open→closed; fallback returns degraded result; retry doesn't double-charge.
- Implement: reuse existing HTTP client, retry settings, or resilience library; keep policy declarative.
Guardrails
- Do not retry non-idempotent money moves without a durable idempotency key.
- Do not set infinite/default-large timeouts — an unbounded wait is an outage.
- Do not retry without backoff/jitter; synchronized retries cause thundering herds.
- Do not mark an op permanently failed when the side effect may have succeeded.
- Do not hide degradation; emit a metric or log so operators see it.