Lifecycle: draft
Resilience and Retry Design
Purpose and scope
Keep failures bounded and recoverable without hiding outages, duplicating effects, or amplifying load. This skill covers dependency interaction and graceful degradation; it does not replace service-specific SLOs, incident response, concurrency review, or capacity planning.
Triggers and prerequisites
Trigger when adding or changing retries, timeouts, polling, backoff, circuit breakers, rate limits, queue handling, fallbacks, cache behavior, or dependency failure paths. Prerequisites: dependency contract, operation idempotence, latency/error distribution, load limits, caller and downstream budgets, user-visible consequences, and recovery/observability capabilities.
Decision criteria
- A timeout bounds waiting; it does not cancel work or undo a side effect unless the dependency guarantees that behavior.
- Retry only failures that are transient and safe to repeat or have an idempotency mechanism. Backoff and jitter must respect the total deadline and downstream capacity.
- A fallback is correct only when its stale, partial, degraded, or unsafe semantics are explicit and acceptable.
- Resilience controls must fail predictably under overload; adding retries without a budget can turn one failure into a larger outage.
Procedure
- Map the call chain, dependency ownership, operation side effects, concurrency, queues, caches, deadlines, and user-visible failure behavior. Identify retry multiplication across layers.
- Classify outcomes: success, permanent failure, transient failure, timeout/unknown outcome, overload, cancellation, partial response, and dependency recovery. Define which are retryable and why.
- Set an end-to-end deadline and budget across callers, retries, backoff, connection, and downstream work. Do not let nested defaults exceed the user or system budget.
- Make repeated operations safe through idempotency keys, deduplication, transactions, conditional writes, or an explicit non-retry policy. Define what happens when the outcome is unknown after a timeout.
- Choose backoff, jitter, retry count, circuit behavior, rate limits, bulkheads, queue limits, and fallback semantics from observed latency, error, and capacity evidence, not folklore.
- Preserve actionable errors and cancellation. Avoid retrying validation, authorization, deterministic failures, or caller cancellation; avoid fallback data that violates freshness, privacy, or correctness requirements.
- Instrument attempts, reasons, delays, outcomes, deadlines, saturation, queue age, circuit state, and fallback use without logging secrets or creating unbounded metrics.
- Test transient/permanent failures, timeouts, slow responses, duplicate delivery, partial side effects, cancellation, overload, recovery, retry storms, concurrent callers, and exhausted budgets.
- Compare load and dependency impact before and after the change. Verify that the system recovers without exceeding downstream limits and that users receive an honest result.
- Record assumptions, retry/fallback contract, budgets, evidence, residual risk, runbook/recovery action, owner, and review trigger.
Examples and counterexamples
Good: A read-only request retries a bounded set of transient transport failures with exponential backoff and jitter, respects an overall deadline, and exposes exhaustion distinctly from an empty result.
Bad: Retry every exception three times, including authorization failures and user cancellation.
Good: A payment operation uses an idempotency key and reconciliation path because a timeout leaves the remote outcome unknown.
Bad: Retry a timed-out charge blindly and risk double payment.
Good: A stale-cache fallback states its freshness limit, records fallback use, and is disabled for operations requiring current authorization or data.
Bad: Return cached sensitive data indefinitely because the dependency is unavailable.
Failure modes and recovery
If idempotence or timeout semantics are unknown, do not add retries; resolve the dependency contract or use a non-retry path. If retry amplification or overload appears, stop retries, shed load, protect the dependency, and follow incident procedures. If a fallback may violate correctness, privacy, or authorization, fail safely and escalate. If a circuit or queue remains open, preserve evidence and define recovery rather than adding unlimited retries.
Validation evidence and provenance
Claims in this skill map to graded findings in docs/research.md:
- Unbounded retries amplify load and cascades propagate through retry feedback loops (Q5, Moderate): Google SRE cascading-failure and overload chapters; convergent practitioner evidence including documented large-scale outages.
- Bounded retries, jitter, budgets, and load shedding as mitigation (Q5, Moderate): same source; specific bound values are deployment-specific decisions.
Source boundary: practitioner references are not controlled studies; graded accordingly.
Confidence: medium. Freshness: review when the research base or dependency-failure evidence changes.
Disconfirmation: evidence that unbounded retry policies do not measurably increase cascading risk in modern systems would require re-grading the bounds requirement.
Related skills and conflicts
Related: concurrency-and-shared-state, evidence-driven-debugging, observability-and-instrumentation, performance-regression-analysis, api-contract-compatibility, secure-coding-review, release-and-rollback-safety, and repository-change-verification. This skill does not authorize infinite retries, hiding failures, duplicating side effects, or weakening correctness for apparent availability.
1---2name: resilience-and-retry-design3description: Design and review timeouts, retries, backoff, circuit breakers, rate limits, bulkheads, and graceful degradation for unreliable dependencies. Use when an AI coding agent changes network calls, queues, workers, caches, or failure handling.4---56Lifecycle: `draft`78# Resilience and Retry Design910## Purpose and scope1112Keep failures bounded and recoverable without hiding outages, duplicating effects, or amplifying load. This skill covers dependency interaction and graceful degradation; it does not replace service-specific SLOs, incident response, concurrency review, or capacity planning.1314## Triggers and prerequisites1516Trigger when adding or changing retries, timeouts, polling, backoff, circuit breakers, rate limits, queue handling, fallbacks, cache behavior, or dependency failure paths. Prerequisites: dependency contract, operation idempotence, latency/error distribution, load limits, caller and downstream budgets, user-visible consequences, and recovery/observability capabilities.1718## Decision criteria1920- A timeout bounds waiting; it does not cancel work or undo a side effect unless the dependency guarantees that behavior.21- Retry only failures that are transient and safe to repeat or have an idempotency mechanism. Backoff and jitter must respect the total deadline and downstream capacity.22- A fallback is correct only when its stale, partial, degraded, or unsafe semantics are explicit and acceptable.23- Resilience controls must fail predictably under overload; adding retries without a budget can turn one failure into a larger outage.2425## Procedure26271. Map the call chain, dependency ownership, operation side effects, concurrency, queues, caches, deadlines, and user-visible failure behavior. Identify retry multiplication across layers.282. Classify outcomes: success, permanent failure, transient failure, timeout/unknown outcome, overload, cancellation, partial response, and dependency recovery. Define which are retryable and why.293. Set an end-to-end deadline and budget across callers, retries, backoff, connection, and downstream work. Do not let nested defaults exceed the user or system budget.304. Make repeated operations safe through idempotency keys, deduplication, transactions, conditional writes, or an explicit non-retry policy. Define what happens when the outcome is unknown after a timeout.315. Choose backoff, jitter, retry count, circuit behavior, rate limits, bulkheads, queue limits, and fallback semantics from observed latency, error, and capacity evidence, not folklore.326. Preserve actionable errors and cancellation. Avoid retrying validation, authorization, deterministic failures, or caller cancellation; avoid fallback data that violates freshness, privacy, or correctness requirements.337. Instrument attempts, reasons, delays, outcomes, deadlines, saturation, queue age, circuit state, and fallback use without logging secrets or creating unbounded metrics.348. Test transient/permanent failures, timeouts, slow responses, duplicate delivery, partial side effects, cancellation, overload, recovery, retry storms, concurrent callers, and exhausted budgets.359. Compare load and dependency impact before and after the change. Verify that the system recovers without exceeding downstream limits and that users receive an honest result.3610. Record assumptions, retry/fallback contract, budgets, evidence, residual risk, runbook/recovery action, owner, and review trigger.3738## Examples and counterexamples3940Good: A read-only request retries a bounded set of transient transport failures with exponential backoff and jitter, respects an overall deadline, and exposes exhaustion distinctly from an empty result.4142Bad: Retry every exception three times, including authorization failures and user cancellation.4344Good: A payment operation uses an idempotency key and reconciliation path because a timeout leaves the remote outcome unknown.4546Bad: Retry a timed-out charge blindly and risk double payment.4748Good: A stale-cache fallback states its freshness limit, records fallback use, and is disabled for operations requiring current authorization or data.4950Bad: Return cached sensitive data indefinitely because the dependency is unavailable.5152## Failure modes and recovery5354If idempotence or timeout semantics are unknown, do not add retries; resolve the dependency contract or use a non-retry path. If retry amplification or overload appears, stop retries, shed load, protect the dependency, and follow incident procedures. If a fallback may violate correctness, privacy, or authorization, fail safely and escalate. If a circuit or queue remains open, preserve evidence and define recovery rather than adding unlimited retries.5556## Validation evidence and provenance57Claims in this skill map to graded findings in [`docs/research.md`](../../docs/research.md):5859- Unbounded retries amplify load and cascades propagate through retry feedback loops (Q5, Moderate): Google SRE cascading-failure and overload chapters; convergent practitioner evidence including documented large-scale outages.60- Bounded retries, jitter, budgets, and load shedding as mitigation (Q5, Moderate): same source; specific bound values are deployment-specific decisions.6162Source boundary: practitioner references are not controlled studies; graded accordingly.6364Confidence: medium. Freshness: review when the research base or dependency-failure evidence changes.6566Disconfirmation: evidence that unbounded retry policies do not measurably increase cascading risk in modern systems would require re-grading the bounds requirement.67## Related skills and conflicts6869Related: `concurrency-and-shared-state`, `evidence-driven-debugging`, `observability-and-instrumentation`, `performance-regression-analysis`, `api-contract-compatibility`, `secure-coding-review`, `release-and-rollback-safety`, and `repository-change-verification`. This skill does not authorize infinite retries, hiding failures, duplicating side effects, or weakening correctness for apparent availability.
ChloeVPin