Failure-Path Review
Happy paths verify themselves: development exercises them, demos exercise them, the first day in production exercises them. Failure paths run only when things go wrong — which is rare, unobserved, and usually concurrent with an incident — so they rot silently. A multi-package review of five broker/durable integrations found exactly this split: every happy path sound, and a defect list consisting entirely of poison handling, redelivery, shutdown, and recovery. Review the unhappy paths as their own deliberate pass, because nothing else will.
The review targets any system that processes work it didn't synchronously receive: message consumers, queue workers, background loops, schedulers, relays, durable workflows.
The passes
Full sweep in references/review-passes.md, with
the checklist form in references/review-checklist.md.
What they all share: failure code only runs when things go wrong, so it is
the code least likely to have been executed even once before it matters.
The seven questions each pass is an instance of:
- What happens to a message that can never succeed? If the answer is "it is retried", the answer is "forever".
- Which failures are retryable, and who decided? A blanket retry on a permanent error is an outage amplifier; a blanket give-up on a transient one is data loss.
- What happens to work in flight when the process dies? Crash-redelivery is the normal case, not the exceptional one.
- Does shutdown drain or abandon? A handler that stops accepting work but never finishes what it holds loses exactly the work that was in progress.
- What grows without bound? Queues, retry counters, in-memory buffers, dead-letter stores.
- What restarts a loop that died? A consumer that exits its own
while True on an unhandled error stays dead, and nothing upstream necessarily notices.
- How would anyone know any of this happened? A failure path with no signal is one that has been working perfectly since the day it was written, as far as anyone can tell.
Verifying the review
Failure paths cannot be verified by running the app — they need forced tests: inject the decode failure, kill the process between effect and ack, force the redelivery, deliver the poison N+1 times, stop the consumer mid-batch. Deterministic fault injection (determinism-by-design) makes these repeatable; the detection branches this review adds are exactly the code that must not be dead (fewer-tests-more-proof), and each fix found follows reproduce-then-fix — forced red first.
Related skills
error-taxonomy — supplies the retryable/terminal classification every pass here consumesreproduce-then-fix — each finding becomes a forced red reproduction before its fixdeterminism-by-design — deterministic fault injection for repeatable failure testsself-audit — its failure-path and cleanup-path passes are the diff-scoped version of this sweep
1---2name: failure-path-review3description: Use when writing or reviewing a consumer, worker, background loop, job runner, retry or backoff path, or shutdown handling; or after an incident involving stuck, lost, or duplicated work. Not for synchronous handlers, and not for diagnosing one live failure.4---56# Failure-Path Review78Happy paths verify themselves: development exercises them, demos exercise them, the first day in production exercises them. Failure paths run only when things go wrong — which is rare, unobserved, and usually concurrent with an incident — so they rot silently. A multi-package review of five broker/durable integrations found exactly this split: *every* happy path sound, and a defect list consisting entirely of poison handling, redelivery, shutdown, and recovery. Review the unhappy paths as their own deliberate pass, because nothing else will.910The review targets any system that processes work it didn't synchronously receive: message consumers, queue workers, background loops, schedulers, relays, durable workflows.1112## The passes1314Full sweep in [references/review-passes.md](references/review-passes.md), with15the checklist form in [references/review-checklist.md](references/review-checklist.md).16What they all share: **failure code only runs when things go wrong,** so it is17the code least likely to have been executed even once before it matters.1819The seven questions each pass is an instance of:2021- **What happens to a message that can never succeed?** If the answer is "it is retried", the answer is "forever".22- **Which failures are retryable, and who decided?** A blanket retry on a permanent error is an outage amplifier; a blanket give-up on a transient one is data loss.23- **What happens to work in flight when the process dies?** Crash-redelivery is the normal case, not the exceptional one.24- **Does shutdown drain or abandon?** A handler that stops accepting work but never finishes what it holds loses exactly the work that was in progress.25- **What grows without bound?** Queues, retry counters, in-memory buffers, dead-letter stores.26- **What restarts a loop that died?** A consumer that exits its own `while True` on an unhandled error stays dead, and nothing upstream necessarily notices.27- **How would anyone know any of this happened?** A failure path with no signal is one that has been working perfectly since the day it was written, as far as anyone can tell.2829## Verifying the review3031Failure paths cannot be verified by running the app — they need forced tests: inject the decode failure, kill the process between effect and ack, force the redelivery, deliver the poison N+1 times, stop the consumer mid-batch. Deterministic fault injection (`determinism-by-design`) makes these repeatable; the detection branches this review adds are exactly the code that must not be dead (`fewer-tests-more-proof`), and each fix found follows `reproduce-then-fix` — forced red first.3233## Related skills3435- `error-taxonomy` — supplies the retryable/terminal classification every pass here consumes36- `reproduce-then-fix` — each finding becomes a forced red reproduction before its fix37- `determinism-by-design` — deterministic fault injection for repeatable failure tests38- `self-audit` — its failure-path and cleanup-path passes are the diff-scoped version of this sweep
morzecrew