Failure Archaeology
The last error is rarely the cause; it is the first symptom loud enough to notice. Work backward to the earliest point where reality started disagreeing with expectation.
Procedure
- Reproduce first. A failure you cannot trigger on demand is not yet understood. Record the exact command and inputs that reproduce it:
python -m app.run --input case.json.
- Read the error text literally for one line — file, line, and message. Do not paraphrase it into a familiar diagnosis; the exact wording usually names the real fault.
- Find the first divergence, not the last crash. For a code regression,
git bisect start; git bisect bad HEAD; git bisect good v1.2.0 and let it find the introducing commit.
- For data-dependent failures, shrink the input by binary search: halve the records; if it still fails, recurse into the failing half until you have the minimal reproducing case.
- Check logs in timestamp order across components, not per component. A message in service A at 10:03:11 usually explains the crash in service B at 10:03:12.
- Distinguish the trigger from the latent bug: a new input may merely expose an ordering or null-handling defect that existed for months. Fix the latent defect, not only the trigger.
- Write a timeline file:
[time] [component] [event] rows, oldest to newest, ending at the observed failure. The row before the first anomaly is your prime suspect.
- Confirm the cause by a falsification test: revert or patch the suspected cause alone and show the failure disappears; then re-introduce it and show it returns.
- Check the environment against last-known-good: dependency versions, env vars, timezone (
node -v, pip freeze | diff - baseline.txt).
- For a regression with no obvious commit, bisect on the input set instead of the code: find the smallest input that flips pass to fail.
- Preserve the failing state before fixing (copy the artefact to
/tmp/fail-case/) so the fix can be re-tested against it.
Pitfalls
- Fixing the final stack frame's error (e.g. a null deref) without asking why the value was null.
- Bisecting code when the divergence is environmental (a dependency version, a clock, a locale).
- Reading one service's log alone and concluding "it just crashed".
- Treating an irreproducible failure as fixed because it did not recur in one run.
- Assuming the cause is the most recent change, when an older latent defect merely surfaced now.
- Concluding "it was a fluke" after one non-reproducing run, then shipping the latent bug.
- Bisecting with a flaky test in the range, so the result points at an innocent commit.
Verification
git bisect reset && git log --oneline -1
# passes when the identified commit, reverted alone, makes the failure disappear
Report to the user: the minimal reproduction, the first divergent event with its timestamp, and the falsification test that confirmed the cause.
1---2name: failure-archaeology3description: Use when something broke and the cause is not obvious from the last error. Locates the first divergent event, not the loudest one, by bisecting state and inputs.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/failure-archaeology6---78# Failure Archaeology910The last error is rarely the cause; it is the first symptom loud enough to notice. Work backward to the earliest point where reality started disagreeing with expectation.1112## Procedure13141. Reproduce first. A failure you cannot trigger on demand is not yet understood. Record the exact command and inputs that reproduce it: `python -m app.run --input case.json`.152. Read the error text literally for one line — file, line, and message. Do not paraphrase it into a familiar diagnosis; the exact wording usually names the real fault.163. Find the first divergence, not the last crash. For a code regression, `git bisect start; git bisect bad HEAD; git bisect good v1.2.0` and let it find the introducing commit.174. For data-dependent failures, shrink the input by binary search: halve the records; if it still fails, recurse into the failing half until you have the minimal reproducing case.185. Check logs in timestamp order across components, not per component. A message in service A at 10:03:11 usually explains the crash in service B at 10:03:12.196. Distinguish the trigger from the latent bug: a new input may merely expose an ordering or null-handling defect that existed for months. Fix the latent defect, not only the trigger.207. Write a timeline file: `[time] [component] [event]` rows, oldest to newest, ending at the observed failure. The row before the first anomaly is your prime suspect.218. Confirm the cause by a falsification test: revert or patch the suspected cause alone and show the failure disappears; then re-introduce it and show it returns.229. Check the environment against last-known-good: dependency versions, env vars, timezone (`node -v`, `pip freeze | diff - baseline.txt`).2310. For a regression with no obvious commit, bisect on the input set instead of the code: find the smallest input that flips pass to fail.2411. Preserve the failing state before fixing (copy the artefact to `/tmp/fail-case/`) so the fix can be re-tested against it.2526## Pitfalls2728- Fixing the final stack frame's error (e.g. a null deref) without asking why the value was null.29- Bisecting code when the divergence is environmental (a dependency version, a clock, a locale).30- Reading one service's log alone and concluding "it just crashed".31- Treating an irreproducible failure as fixed because it did not recur in one run.32- Assuming the cause is the most recent change, when an older latent defect merely surfaced now.33- Concluding "it was a fluke" after one non-reproducing run, then shipping the latent bug.34- Bisecting with a flaky test in the range, so the result points at an innocent commit.3536## Verification3738 git bisect reset && git log --oneline -139 # passes when the identified commit, reverted alone, makes the failure disappear4041Report to the user: the minimal reproduction, the first divergent event with its timestamp, and the falsification test that confirmed the cause.