Systematic Debugging

Systematic debugging

The symptom is where the failure surfaced, not where it came from. Editing the line the report names is how a bug gets moved rather than fixed.

When this fires

• A defect, crash, wrong output, corrupted state or failing test whose cause is not yet known.
• A second attempt at a fix, after the first one did not hold or held for reasons unknown.
• Before any edit whose only justification is that the code looks suspicious.

It does not fire during a live outage — mitigation comes first and the explanation comes after — and it does not fire when the mechanism is already established and only the patch remains.

Procedure

1. State the failure precisely. Expected versus actual, the exact input, the version and environment, and when it started. "Login is broken" is not a failure statement. "POST /login returns 500 on staging for accounts created before the column was added" is one.
2. Reproduce it before touching anything. Record the exact command or step list and its output. If it will not reproduce, that is the finding — report how you tried and how often it fires. Do not proceed to a fix whose effect you have no way to observe.
3. Capture evidence before theorising. The full stack trace, not the last frame. The log lines either side of the failure. The failing input. What changed recently in the code and in its dependencies. Read all of it. A large share of bugs are already named in evidence nobody read.
4. Bound it. Which inputs fail and which succeed, which callers, which versions, which environments. A failure confined to one input class is already half diagnosed.
5. Write hypotheses down, ranked. Each one must predict something you have not yet looked at — "if it is the stale cache, this key holds the old value" — and then you go and look. A hypothesis that only explains what you have already seen cannot be tested.
6. Test one at a time, cheapest first. Change one variable per experiment. A log line, a breakpoint or a query beats a speculative edit: reading adds no new variables, editing does.
7. Isolate to the smallest failing case. Bisect whichever axis you can halve — the history, the input, the configuration, the code path. Stop when removing one more thing makes it pass.
8. Name the causal mechanism in one sentence before you edit. If the best you have is "something about async timing", you are not ready to fix; you are ready to keep isolating.
9. Fix the cause where all callers route through it. Grep every caller of the function you are about to change. Patching only the path the ticket named leaves the siblings broken; one guard in the shared function is both the smaller diff and the actual fix.
10. Prove it. Re-run the reproduction from step 2. Add a regression test that fails on the old code (regression-testing). Run the surrounding suite to see the fix broke nothing adjacent.
11. Remove your instrumentation — debug prints, extra logging, loosened timeouts. Anything worth keeping is kept deliberately and said out loud, not left in the diff by accident.

Anti-patterns — the random-patch loop

The loop this skill exists to break: change something plausible, see whether the symptom moves, repeat. Every round adds a variable, and the run that finally passes is indistinguishable from coincidence.

• Fix by disappearance. The symptom stopped after five edits, so one of them worked. You do not know which, or whether it merely became less frequent.
• A delay is not a fix. A sleep or a retry makes a race rarer. Report it as a mitigation with the race still unexplained, never as a root cause.
• Suppression is not a fix. A broad catch, a disabled assertion, a loosened check — the failure is now invisible rather than absent.
• Anchoring. The first plausible story gets all the evidence read into it. Ask what the evidence would look like if that story were wrong, then check for that.
• Rewriting the region. A rewrite that happens to pass has not been diagnosed; it has replaced a known bug with unknown ones.

Checklist

• Reproduction captured as an exact command or step list, and run before any edit
• Full stack trace and surrounding logs read
• Hypotheses written down, each predicting something unobserved
• Smallest failing case in hand
• Causal mechanism stated in one sentence
• Every caller of the changed code checked, not only the reported path
• Original reproduction re-run after the fix
• Regression test confirmed to fail against the old code
• Instrumentation removed, or deliberately kept and said so

Failure handling

• Will not reproduce — report it as not reproduced, with the conditions tried. If a low-risk speculative fix ships anyway, label it unverified in plain words. An intermittent failure announced as fixed is worse than one announced as intermittent.
• Reproduction is slow or manual — build a cheap one first, a script or a failing test. You are going to run it dozens of times.
• No logs, no traces, nothing to read — adding instrumentation is a step, not a fix. Record the observability gap as a finding in its own right.
• Every hypothesis falls — widen the frame to what you assumed and never checked: the environment, a dependency version, the data, the clock, another process, the build itself.
• Testing it needs something destructive or outward-facing — deleting data, writing to a shared environment, calling a live third party, deploying. Stop and ask. Reproduce against a local or throwaway copy instead.

Evidence to report

The reproduction command and its output before the fix; the evidence that ruled each hypothesis in or out; the causal mechanism in one sentence; the diff; the regression test with proof it fails without the fix; and the reproduction re-run and now passing. Name what remains untested and any hypothesis you could not eliminate. A fix that was written and a fix that was proven are different claims — "fixed" without the re-run reproduction is the first one wearing the second one's clothes.