Systematic debugging
The rule
No fix before the cause is proven. A change that makes a symptom disappear without an
explanation has not fixed anything — it has moved the failure somewhere you are not looking.
Method
- Reproduce it deterministically. If you cannot make it happen on demand, you cannot know when
it is fixed. Intermittent means you have not found the variable yet — order, timing, state,
environment, data.
- Narrow the blast radius. Bisect: which commit, which input, which branch, which layer. Halve
the search space with each step rather than reading everything.
- State a hypothesis that can be wrong. "The cache returns stale rows after a write" is a
hypothesis. "Something is wrong with caching" is not.
- Test the hypothesis directly — a log line, a breakpoint, a probe. Prove it, do not infer it.
- Explain the whole symptom. If your cause explains the error but not why it started Tuesday,
you have found a bug, not the bug.
- Fix, then verify by reverting. Put the bug back and confirm the test fails again. This is the
step people skip, and it is the one that proves causation rather than coincidence.
Anti-patterns
- Shotgun changes — altering several things at once. Now you cannot attribute the fix.
- "Probably a flake." Not a diagnosis. A test that fails intermittently is reporting a real
race, ordering dependency, or shared-state leak.
- Fixing the symptom — catching the exception, adding a retry, widening a timeout — without
knowing what threw it.
- Trusting the error message's location. Where it surfaced is rarely where it originated.
Never
- Close a bug without a reproduction that failed before the fix and passes after it.
- Deploy a change to production to find out whether it works. Reproduce somewhere you can observe first.
- Change the environment and the code in the same step.
- Leave a debugging aid in the code as the fix — a widened timeout, a disabled check, a swallowed exception.
Return contract
State the reproduction, the proven cause, the fix, and the verification that the fix addresses that
cause specifically. Name anything you ruled out and how.
1---2name: systematic-debugging3description: Finds the root cause of a bug, test failure, or unexpected behavior before proposing any fix. Use this whenever something is broken and the cause is not yet proven — a failing test, a production error, intermittent behavior, or a symptom that appeared after a change. Also use when a fix has been attempted and did not work, or when the same bug keeps coming back.4---56# Systematic debugging78## The rule910**No fix before the cause is proven.** A change that makes a symptom disappear without an11explanation has not fixed anything — it has moved the failure somewhere you are not looking.1213## Method14151. **Reproduce it deterministically.** If you cannot make it happen on demand, you cannot know when16 it is fixed. Intermittent means you have not found the variable yet — order, timing, state,17 environment, data.182. **Narrow the blast radius.** Bisect: which commit, which input, which branch, which layer. Halve19 the search space with each step rather than reading everything.203. **State a hypothesis that can be wrong.** "The cache returns stale rows after a write" is a21 hypothesis. "Something is wrong with caching" is not.224. **Test the hypothesis directly** — a log line, a breakpoint, a probe. Prove it, do not infer it.235. **Explain the whole symptom.** If your cause explains the error but not why it started Tuesday,24 you have found *a* bug, not *the* bug.256. **Fix, then verify by reverting.** Put the bug back and confirm the test fails again. This is the26 step people skip, and it is the one that proves causation rather than coincidence.2728## Anti-patterns2930- **Shotgun changes** — altering several things at once. Now you cannot attribute the fix.31- **"Probably a flake."** Not a diagnosis. A test that fails intermittently is reporting a real32 race, ordering dependency, or shared-state leak.33- **Fixing the symptom** — catching the exception, adding a retry, widening a timeout — without34 knowing what threw it.35- **Trusting the error message's location.** Where it surfaced is rarely where it originated.3637## Never3839- Close a bug without a reproduction that failed before the fix and passes after it.40- Deploy a change to production to find out whether it works. Reproduce somewhere you can observe first.41- Change the environment and the code in the same step.42- Leave a debugging aid in the code as the fix — a widened timeout, a disabled check, a swallowed exception.4344## Return contract4546State the reproduction, the proven cause, the fix, and the verification that the fix addresses that47cause specifically. Name anything you ruled out and how.
cbrock84