Handling failures
REQUIRED BACKGROUND: the principal-engineering skill.
Overview
A swallowed error is a bug with its evidence destroyed. Core contract: every failure path does exactly one of three things, and all three are loud. The system that looks healthy while serving wrong data is worse than the one that crashes, because the crash gets fixed today and the silent wrong output gets discovered in an audit.
The contract
Every catch and failure path either:
- Logs at WARN or ERROR and rethrows, or
- Logs and returns a TYPED failure the caller must handle (a result type, a sealed error, a status the compiler or contract forces downstream code to acknowledge), or
- Logs and enters an explicitly documented degraded mode (the degradation is named in the code and its documentation, and something observable says the system is degraded).
Forbidden, no exceptions:
- Bare catch-and-continue.
- Catch-and-return-default (empty list, null, zero, cached copy) that masks the failure.
?? fallback and its cousins where the fallback hides that the primary failed. A fallback is acceptable only when the absence is ALSO surfaced loudly elsewhere.
Each of these converts a detectable failure into silent wrong output.
Corollaries
- A missing required entry fails loud. Something absent from a registry, config, or catalog is a build or startup failure, never a silent default; otherwise the single source quietly becomes optional.
- Error states are visible. A workflow must not appear healthy while failing; surface the error state in the UI, the metrics, or the logs someone actually watches.
- Operator-facing remediation is specific. An operator reads the error message mid-incident; "connection failed, check REDIS_URL and whether redis responds to PING" beats "an error occurred" by the length of the outage.
- Retries are bounded and observable.
- Replays of side-effecting operations are idempotent, or they multiply the damage: a retry queue replaying charges is how an outage becomes a refund program.
- Degraded modes have a bound. Skip-and-continue needs the explicit threshold where degradation becomes abort, as a named, operator-tunable constant. The guard must exist; its value is a judgment call to make with the owner, and an unbounded degraded mode is a slow-motion swallow.
- On failure paths, observability is part of the minimum, not gold-plating. The log line, the counter, and the alert ship with the fix; a failure path without them is the silent swallow with better intentions.
Touching existing swallows
Code you are editing that already swallows: fix it as part of the work, or explicitly flag it as owed with what it hides. Leaving it silently is endorsing it. In review, a NEW silent swallow is an automatic BLOCKER; a pre-existing one you touched and left unflagged is a WARNING against the change.
Common mistakes
- "It should never happen" as a reason to swallow. Paths that should never happen are exactly the ones that need a loud alarm when they do.
- Logging at DEBUG and calling it handled. If nobody sees it in production, it is a swallow with extra steps.
- Catching broad (
Exception, catch {}) to handle narrow. The unexpected failure rides in with the expected one and dies silently beside it.
- A degraded mode nobody documented. Degradation that only the author knows about is an outage the operator cannot diagnose.
- Making the test pass by defaulting the failure. The test goes green; the defect graduates to production.
1---2name: handling-failures3description: Use when writing or touching any error path, catch block, fallback, default value, retry, or degraded mode - in any language, any repo. Encodes the no-silent-swallows contract and the fail-loud discipline. Use whenever an exception is about to be caught, a null is about to get a default, or a failure could pass unnoticed, even if the goal is "just make it not crash".4---5
6# Handling failures
7
8**REQUIRED BACKGROUND:** the `principal-engineering` skill.
9
10## Overview
11
12A swallowed error is a bug with its evidence destroyed. Core contract: **every failure path does exactly one of three things, and all three are loud.** The system that looks healthy while serving wrong data is worse than the one that crashes, because the crash gets fixed today and the silent wrong output gets discovered in an audit.
13
14## The contract
15
16Every catch and failure path either:
17
181. **Logs at WARN or ERROR and rethrows**, or
192. **Logs and returns a TYPED failure the caller must handle** (a result type, a sealed error, a status the compiler or contract forces downstream code to acknowledge), or
203. **Logs and enters an explicitly documented degraded mode** (the degradation is named in the code and its documentation, and something observable says the system is degraded).
21
22Forbidden, no exceptions:
23
24- Bare catch-and-continue.
25- Catch-and-return-default (empty list, null, zero, cached copy) that masks the failure.
26- `?? fallback` and its cousins where the fallback hides that the primary failed. A fallback is acceptable only when the absence is ALSO surfaced loudly elsewhere.
27
28Each of these converts a detectable failure into silent wrong output.
29
30## Corollaries
31
32- **A missing required entry fails loud.** Something absent from a registry, config, or catalog is a build or startup failure, never a silent default; otherwise the single source quietly becomes optional.
33- **Error states are visible.** A workflow must not appear healthy while failing; surface the error state in the UI, the metrics, or the logs someone actually watches.
34- **Operator-facing remediation is specific.** An operator reads the error message mid-incident; "connection failed, check REDIS_URL and whether redis responds to PING" beats "an error occurred" by the length of the outage.
35- **Retries are bounded and observable.**
36- **Replays of side-effecting operations are idempotent**, or they multiply the damage: a retry queue replaying charges is how an outage becomes a refund program.
37- **Degraded modes have a bound.** Skip-and-continue needs the explicit threshold where degradation becomes abort, as a named, operator-tunable constant. The guard must exist; its value is a judgment call to make with the owner, and an unbounded degraded mode is a slow-motion swallow.
38- **On failure paths, observability is part of the minimum**, not gold-plating. The log line, the counter, and the alert ship with the fix; a failure path without them is the silent swallow with better intentions.
39
40## Touching existing swallows
41
42Code you are editing that already swallows: fix it as part of the work, or explicitly flag it as owed with what it hides. Leaving it silently is endorsing it. In review, a NEW silent swallow is an automatic BLOCKER; a pre-existing one you touched and left unflagged is a WARNING against the change.
43
44## Common mistakes
45
46- "It should never happen" as a reason to swallow. Paths that should never happen are exactly the ones that need a loud alarm when they do.
47- Logging at DEBUG and calling it handled. If nobody sees it in production, it is a swallow with extra steps.
48- Catching broad (`Exception`, `catch {}`) to handle narrow. The unexpected failure rides in with the expected one and dies silently beside it.
49- A degraded mode nobody documented. Degradation that only the author knows about is an outage the operator cannot diagnose.
50- Making the test pass by defaulting the failure. The test goes green; the defect graduates to production.
riekelt