Reliability And Failure Audit
Definition
Confirm the system behaves predictably when things go wrong. Production readiness requires recovery paths, not only happy paths.
Questions To Ask
- What dependencies can fail, slow down, or return partial results?
- Which operations must be idempotent?
- What retries, timeouts, and cancellation rules exist?
- What concurrent or duplicate requests can happen?
- How does the system recover after partial failure?
Existing Project Comparison
- Inspect queues, jobs, webhooks, network calls, transactions, locks, idempotency keys, retry policies, and error handling.
- Compare failure behavior against requirements and operational runbooks.
- Look for avoidable special cases and errors that can be designed out of existence.
Suggestive Plan
- Map production dependencies and failure modes.
- Inspect retry, timeout, idempotency, and concurrency handling.
- Run or review failure-injection, edge-case, and recovery tests.
- Classify missing recovery behavior as blocker or accepted risk.
- Update
PRODUCTION_READINESS.md.
Example
For payment webhooks, verify duplicate events do not double-charge, provider timeouts retry safely, and partial database writes recover or roll back.
Vocabulary
- Idempotency: repeated request has the same effect as one request.
- Partial failure: some steps succeed while others fail.
- Recovery path: behavior that restores a safe state after failure.
- Timeout: maximum wait before abandoning or retrying work.
Expected Outcome
Produce a reliability audit with dependency risks, failure-mode evidence, recovery gaps, blockers, accepted risks, and required fixes.
1---2name: reliability-failure-audit3description: Audit reliability and failure readiness before production. Use when testing dependency failures, retries, timeouts, duplicate requests, concurrency, partial failures, idempotency, recovery behavior, and error handling.4---56# Reliability And Failure Audit78## Definition910Confirm the system behaves predictably when things go wrong. Production readiness requires recovery paths, not only happy paths.1112## Questions To Ask1314- What dependencies can fail, slow down, or return partial results?15- Which operations must be idempotent?16- What retries, timeouts, and cancellation rules exist?17- What concurrent or duplicate requests can happen?18- How does the system recover after partial failure?1920## Existing Project Comparison2122- Inspect queues, jobs, webhooks, network calls, transactions, locks, idempotency keys, retry policies, and error handling.23- Compare failure behavior against requirements and operational runbooks.24- Look for avoidable special cases and errors that can be designed out of existence.2526## Suggestive Plan27281. Map production dependencies and failure modes.292. Inspect retry, timeout, idempotency, and concurrency handling.303. Run or review failure-injection, edge-case, and recovery tests.314. Classify missing recovery behavior as blocker or accepted risk.325. Update `PRODUCTION_READINESS.md`.3334## Example3536For payment webhooks, verify duplicate events do not double-charge, provider timeouts retry safely, and partial database writes recover or roll back.3738## Vocabulary3940- Idempotency: repeated request has the same effect as one request.41- Partial failure: some steps succeed while others fail.42- Recovery path: behavior that restores a safe state after failure.43- Timeout: maximum wait before abandoning or retrying work.4445## Expected Outcome4647Produce a reliability audit with dependency risks, failure-mode evidence, recovery gaps, blockers, accepted risks, and required fixes.
harsh817