Investigate Root Cause
Overview
A four-phase debugging workflow that forces an engineer to gather evidence and write
down a hypothesis before changing any code. The skill exists because the most
common debugging failure isn't a missing technique — it's the engineer skipping past
the error message, forming a vague mental theory, and patching the symptom. Every
phase here produces an artifact you could paste into a code review. If you can't
produce the artifact, you haven't done the phase. The skill is for senior ICs and
tech leads who already know how to debug; what it adds is the discipline to refuse
to fix what you don't yet understand.
When to Use
- A test is failing and you don't already know why
- An error message appeared that you cannot immediately point to a line of code for
- A reproduction is intermittent (sometimes passes, sometimes fails)
- A previously passing system started failing after no obvious cause
- Production is misbehaving and the cause isn't in the most recent commit
- You catch yourself about to write a fix while still uncertain why the bug happens
When NOT to Use
- The error message names a missing import, typo, or syntax error and the fix is one
character. Just fix it.
- The runbook for this exact failure exists and the documented fix has been applied
before. Follow the runbook.
- The "bug" is a config value that needs flipping in an environment variable. Flip it.
Process
Four phases. Each phase has a gate. You do not advance until the gate's evidence
artifact exists.
Phase 1: Gather
Goal: Surface every fact that already exists about this bug, before forming any
theory.
Inputs: A bug report, a failing test, an error message, or a complaint
("it doesn't work").
Actions:
- Capture the literal error. Copy the full text of the error message and the
complete stack trace. Do not paraphrase. If there is no error message, write down
the exact observed-vs-expected behavior in one sentence each.
- Find the reproduction. Run the failing scenario yourself. Record the exact
command, environment, and inputs. If you cannot reproduce it, that is the bug to
investigate first — go to Step 3 and Step 4 and stay in Phase 1 until you can.
- Read recent history. Run
git log --oneline -30 and read the last 30 commits.
Note which commits touch files in the stack trace.
- Collect logs. Pull logs around the failure window. If structured logs exist,
filter to the request or session that hit the bug. If not, raise the verbosity
and re-run the reproduction.
- Look at the data. If the bug involves a record, fetch the record. If it
involves a query, run the query. If it involves a request body, capture the body.
Output: A short text block titled Phase 1: Gather containing the literal error
text, the exact reproducer command, the relevant commit hashes, log excerpts, and
data values. Pasted into a scratch file or PR description.
Phase 2: Hypothesize
Goal: Convert evidence into a single specific written hypothesis. One.
Inputs: The Phase 1 artifact.
Actions:
- Find a working comparison. Locate the closest equivalent code path that
succeeds. Read it. Note the differences.
- Identify the smallest difference that matters. Configuration, data shape,
environment, timing, or contract. Name it.
- Write the hypothesis as one sentence in this exact form:
The bug occurs because [X] causes [Y] when [Z].
No "I think." No "maybe." If you can't fill all three slots, return to Phase 1.
Output: A one-sentence hypothesis added under Phase 2: Hypothesize. Plus the
file:line citation of the working comparison code.
Phase 3: Test
Goal: Prove or disprove the hypothesis with a single deliberate change.
Inputs: The hypothesis from Phase 2.
Actions:
- Design the smallest test of the hypothesis. Often this is a one-line
print / console.error / breakpoint at the line where you predicted the
anomaly happens, NOT a fix.
- Run it. Capture the output. Record what you saw with the same rigor as
Phase 1's literal error capture.
- Decide: does the output confirm or refute the hypothesis?
- Confirm: advance to Phase 4.
- Refute: return to Phase 2 with the new evidence. Update the hypothesis.
Do not start patching.
Output: Under Phase 3: Test, the exact instrumentation used, the output
captured, and a one-line verdict: Hypothesis confirmed | Hypothesis refuted → return to Phase 2.
Phase 4: Prove
Goal: A fix that addresses the cause, with a regression test that pins it.
Inputs: A confirmed hypothesis.
Actions:
- Write a failing test that captures the bug. The test fails on
main and
passes after the fix. It exercises the cause, not the symptom.
- Make the smallest change that makes the test pass. Single targeted fix at
the cause. Do not bundle other improvements.
- Run the failing test. Confirm it passes.
- Run the full test suite. Confirm green.
- Run the original reproduction from Phase 1. Confirm fixed.
Output: Under Phase 4: Prove, paste:
- Failing test name and location
- Test runner output before fix (red)
- Test runner output after fix (green)
- Full-suite output (green)
- Original Phase 1 reproducer output (now passing)
Rationalizations
| Excuse |
Why it sounds reasonable |
Why it's wrong |
What to do instead |
| "I think I see the problem — let me just patch it." |
The fix often is small once you understand it. The instinct that you "see it" feels like signal. |
If you were right, you wouldn't need a hypothesis. The "I see it" feeling is pattern-matching on similar bugs you've seen before, and pattern-matching has a high false-positive rate on real systems. The patches that ship from this state usually fix the symptom one observation downstream of the cause. |
Phase 2 anyway. Write the hypothesis sentence. If it really is obvious, this takes 60 seconds. If you can't write the sentence, you didn't actually see it. |
| "Can't reproduce locally — must be a flake." |
Flakes do exist, and chasing a non-reproducer wastes time. |
"Flake" is what we call a bug whose trigger condition we haven't found yet. Closing a ticket as "flaky" hands the bug to the next person who hits it, plus accumulated mystery. The trigger is real; you just don't know it yet. |
Treat "can't reproduce" as the bug. Phase 1, Step 2: list every difference between your environment and the failing one (timezone, locale, clock skew, parallelism, container vs host, data size, prior test state). Bisect on differences. |
| "It worked before the last deploy — it's the deploy." |
Recent deploys do cause regressions, and git bisect is real evidence. |
"It's the deploy" without bisect is folklore. The deploy may have shifted timing, exposed a latent bug, or changed something orthogonal. Skipping bisect means the fix may also be folklore. |
Run git bisect between the last known good and the first known bad. Cite the actual offending commit hash in the hypothesis. |
| "It's probably a race condition." / "Must be caching." |
These categories explain a lot of intermittent bugs. |
Naming a category is not a hypothesis. "Race condition" doesn't tell you which two operations race or what the interleaving is. Until you can write [X] causes [Y] when [Z] with the actual operations and ordering, you're labeling, not investigating. |
Phase 2 with concrete operations: which thread/request reads, which writes, what happens when the write lands during the read. Same shape for caching: which key, which TTL, what stale value, who serves it. |
| "Let me wrap it in a try/catch and move on." |
Defensive coding is a real practice, and silencing exceptions does keep the surface stable. |
Catching the exception that resulted from the bug doesn't fix the bug — it hides the evidence the next investigator needs. The system continues to be wrong, just quieter. The next failure will be downstream and harder to trace. |
If a try/catch is appropriate for known failure modes, fine — but only after the cause is understood. The catch goes in Phase 4 with a hypothesis-confirmed reason for tolerating that failure mode. Otherwise you are masking. |
| "I'll add some logs and check it tomorrow." |
Adding logging is a real Phase 1 action. |
The trap is the "tomorrow" part — logs that get added without a written hypothesis, drift in the codebase as cruft, and never get analyzed because by tomorrow the urgent thing has shifted. The investigation gets put down without a marker. |
Add logs, but inside Phase 1 with a written reason: "logging X to test whether Y occurs before Z." Set a calendar reminder for the analysis. If you won't analyze tomorrow, don't add the logs. |
| "The error message is misleading — the real bug is somewhere else." |
Sometimes errors do surface far from their cause. |
"The error is misleading" said before Phase 1's literal capture is the engineer dismissing evidence they haven't read carefully yet. The error message is data; "misleading" is a story you tell about data. Read the data first. |
Paste the literal error in Phase 1. If the message names a file:line, look at that file:line before declaring the message is wrong. Most "misleading" errors are accurate; the engineer was holding a wrong mental model of which code runs first. |
Evidence Requirements
Every phase has a gate. If the gate's artifact does not exist, that phase has not
been completed.
| Checkpoint |
Required artifact |
What "no evidence" looks like |
| End of Phase 1 |
Literal error text + reproducer command + relevant commit hashes pasted in a Phase 1: Gather block |
"I read through the code and I'm pretty sure it's in the auth module." |
| End of Phase 2 |
One sentence in form The bug occurs because [X] causes [Y] when [Z] |
"It's probably a race condition somewhere in the request lifecycle." |
| End of Phase 3 |
Captured output from a deliberate test of the hypothesis (instrumentation OR experiment), plus a confirm/refute verdict |
"Yeah I tried a thing and it seemed to work." |
| End of Phase 4 |
Failing test (red), passing test after fix (green), full suite (green), original reproducer (fixed) — all four pasted |
"Tests pass on my machine." |
If you can't paste it, you haven't done it. Stop.
Red Flags
Concrete observations that mean STOP and reassess.
- You've changed the same line three or more different ways in the last hour. You
don't have a working hypothesis; you're guessing.
- You added a
try/catch, if err == nil, or test-skip whose justification is
"to make the test pass." That's masking, not fixing.
- The hypothesis sentence is missing the
when [Z] clause. You don't know the
trigger condition. The fix will be partial.
- Three consecutive fix attempts have failed. The bug is architectural, not local.
Escalate or rescope.
- You're about to ship a fix you cannot explain to the next reviewer in one
sentence. The reviewer won't accept it; you shouldn't either.
- The failing test you wrote in Phase 4 doesn't actually fail on
main without
the fix. It tests something tangential. Rewrite it.
References
- John Allspaw & Richard Cook, How Complex Systems Fail (Cognitive Technologies
Laboratory, 1998) — point #5 ("Complex systems run in degraded mode") and point
#14 ("Change introduces new forms of failure"). Use these to resist the "it
worked before the deploy" reflex; the post-deploy failure is often a latent
problem made visible, not the deploy itself.
- Site Reliability Engineering, Beyer et al. (Google, O'Reilly 2016), Chapter 12
"Effective Troubleshooting" — defines the diagnose-test-fix loop this skill's
Phases 2-4 implement, and explicitly warns against the "I know what's wrong"
pattern handled in the Rationalizations table.
1---2name: investigate-root-cause3description: Use when encountering ANY bug, error, test failure, or unexpected behavior. Activate for keywords like "bug", "error", "failing", "broken", "doesn't work", "unexpected", "crash", "exception", "TypeError", "undefined", stack traces, or any error message. Also trigger when tests fail unexpectedly, when behavior differs from expectations, when investigating production incidents, or when flaky/intermittent issues appear. Investigation produces evidence and a written hypothesis before any fix is attempted. Always investigate root cause before proposing fixes -- never guess at solutions.4---56# Investigate Root Cause78## Overview910A four-phase debugging workflow that forces an engineer to gather evidence and write11down a hypothesis *before* changing any code. The skill exists because the most12common debugging failure isn't a missing technique — it's the engineer skipping past13the error message, forming a vague mental theory, and patching the symptom. Every14phase here produces an artifact you could paste into a code review. If you can't15produce the artifact, you haven't done the phase. The skill is for senior ICs and16tech leads who already know how to debug; what it adds is the discipline to refuse17to fix what you don't yet understand.1819## When to Use2021- A test is failing and you don't already know why22- An error message appeared that you cannot immediately point to a line of code for23- A reproduction is intermittent (sometimes passes, sometimes fails)24- A previously passing system started failing after no obvious cause25- Production is misbehaving and the cause isn't in the most recent commit26- You catch yourself about to write a fix while still uncertain why the bug happens2728## When NOT to Use2930- The error message names a missing import, typo, or syntax error and the fix is one31 character. Just fix it.32- The runbook for this exact failure exists and the documented fix has been applied33 before. Follow the runbook.34- The "bug" is a config value that needs flipping in an environment variable. Flip it.3536## Process3738Four phases. Each phase has a gate. You do not advance until the gate's evidence39artifact exists.4041### Phase 1: Gather4243**Goal:** Surface every fact that already exists about this bug, before forming any44theory.4546**Inputs:** A bug report, a failing test, an error message, or a complaint47("it doesn't work").4849**Actions:**50511. **Capture the literal error.** Copy the full text of the error message and the52 complete stack trace. Do not paraphrase. If there is no error message, write down53 the exact observed-vs-expected behavior in one sentence each.542. **Find the reproduction.** Run the failing scenario yourself. Record the exact55 command, environment, and inputs. If you cannot reproduce it, that is the bug to56 investigate first — go to Step 3 and Step 4 and stay in Phase 1 until you can.573. **Read recent history.** Run `git log --oneline -30` and read the last 30 commits.58 Note which commits touch files in the stack trace.594. **Collect logs.** Pull logs around the failure window. If structured logs exist,60 filter to the request or session that hit the bug. If not, raise the verbosity61 and re-run the reproduction.625. **Look at the data.** If the bug involves a record, fetch the record. If it63 involves a query, run the query. If it involves a request body, capture the body.6465**Output:** A short text block titled `Phase 1: Gather` containing the literal error66text, the exact reproducer command, the relevant commit hashes, log excerpts, and67data values. Pasted into a scratch file or PR description.6869### Phase 2: Hypothesize7071**Goal:** Convert evidence into a single specific written hypothesis. One.7273**Inputs:** The Phase 1 artifact.7475**Actions:**76771. **Find a working comparison.** Locate the closest equivalent code path that78 succeeds. Read it. Note the differences.792. **Identify the smallest difference that matters.** Configuration, data shape,80 environment, timing, or contract. Name it.813. **Write the hypothesis as one sentence in this exact form:**82 `The bug occurs because [X] causes [Y] when [Z].`83 No "I think." No "maybe." If you can't fill all three slots, return to Phase 1.8485**Output:** A one-sentence hypothesis added under `Phase 2: Hypothesize`. Plus the86file:line citation of the working comparison code.8788### Phase 3: Test8990**Goal:** Prove or disprove the hypothesis with a single deliberate change.9192**Inputs:** The hypothesis from Phase 2.9394**Actions:**95961. **Design the smallest test of the hypothesis.** Often this is a one-line97 `print` / `console.error` / breakpoint at the line where you predicted the98 anomaly happens, NOT a fix.992. **Run it. Capture the output.** Record what you saw with the same rigor as100 Phase 1's literal error capture.1013. **Decide:** does the output confirm or refute the hypothesis?102 - **Confirm:** advance to Phase 4.103 - **Refute:** return to Phase 2 with the new evidence. Update the hypothesis.104 Do not start patching.105106**Output:** Under `Phase 3: Test`, the exact instrumentation used, the output107captured, and a one-line verdict: `Hypothesis confirmed | Hypothesis refuted →108return to Phase 2`.109110### Phase 4: Prove111112**Goal:** A fix that addresses the cause, with a regression test that pins it.113114**Inputs:** A confirmed hypothesis.115116**Actions:**1171181. **Write a failing test that captures the bug.** The test fails on `main` and119 passes after the fix. It exercises the cause, not the symptom.1202. **Make the smallest change that makes the test pass.** Single targeted fix at121 the cause. Do not bundle other improvements.1223. **Run the failing test. Confirm it passes.**1234. **Run the full test suite. Confirm green.**1245. **Run the original reproduction from Phase 1. Confirm fixed.**125126**Output:** Under `Phase 4: Prove`, paste:127- Failing test name and location128- Test runner output before fix (red)129- Test runner output after fix (green)130- Full-suite output (green)131- Original Phase 1 reproducer output (now passing)132133## Rationalizations134135| Excuse | Why it sounds reasonable | Why it's wrong | What to do instead |136|---|---|---|---|137| "I think I see the problem — let me just patch it." | The fix often is small once you understand it. The instinct that you "see it" feels like signal. | If you were right, you wouldn't need a hypothesis. The "I see it" feeling is pattern-matching on similar bugs you've seen before, and pattern-matching has a high false-positive rate on real systems. The patches that ship from this state usually fix the *symptom* one observation downstream of the cause. | Phase 2 anyway. Write the hypothesis sentence. If it really is obvious, this takes 60 seconds. If you can't write the sentence, you didn't actually see it. |138| "Can't reproduce locally — must be a flake." | Flakes do exist, and chasing a non-reproducer wastes time. | "Flake" is what we call a bug whose trigger condition we haven't found yet. Closing a ticket as "flaky" hands the bug to the next person who hits it, plus accumulated mystery. The trigger is real; you just don't know it yet. | Treat "can't reproduce" as the bug. Phase 1, Step 2: list every difference between your environment and the failing one (timezone, locale, clock skew, parallelism, container vs host, data size, prior test state). Bisect on differences. |139| "It worked before the last deploy — it's the deploy." | Recent deploys do cause regressions, and `git bisect` is real evidence. | "It's the deploy" without bisect is folklore. The deploy may have shifted timing, exposed a latent bug, or changed something orthogonal. Skipping bisect means the fix may also be folklore. | Run `git bisect` between the last known good and the first known bad. Cite the actual offending commit hash in the hypothesis. |140| "It's probably a race condition." / "Must be caching." | These categories explain a lot of intermittent bugs. | Naming a category is not a hypothesis. "Race condition" doesn't tell you which two operations race or what the interleaving is. Until you can write `[X] causes [Y] when [Z]` with the actual operations and ordering, you're labeling, not investigating. | Phase 2 with concrete operations: which thread/request reads, which writes, what happens when the write lands during the read. Same shape for caching: which key, which TTL, what stale value, who serves it. |141| "Let me wrap it in a try/catch and move on." | Defensive coding is a real practice, and silencing exceptions does keep the surface stable. | Catching the exception that resulted from the bug doesn't fix the bug — it hides the evidence the next investigator needs. The system continues to be wrong, just quieter. The next failure will be downstream and harder to trace. | If a try/catch is appropriate for *known* failure modes, fine — but only after the cause is understood. The catch goes in Phase 4 *with* a hypothesis-confirmed reason for tolerating that failure mode. Otherwise you are masking. |142| "I'll add some logs and check it tomorrow." | Adding logging is a real Phase 1 action. | The trap is the "tomorrow" part — logs that get added without a written hypothesis, drift in the codebase as cruft, and never get analyzed because by tomorrow the urgent thing has shifted. The investigation gets put down without a marker. | Add logs, but inside Phase 1 with a written reason: "logging X to test whether Y occurs before Z." Set a calendar reminder for the analysis. If you won't analyze tomorrow, don't add the logs. |143| "The error message is misleading — the real bug is somewhere else." | Sometimes errors do surface far from their cause. | "The error is misleading" said *before* Phase 1's literal capture is the engineer dismissing evidence they haven't read carefully yet. The error message is data; "misleading" is a story you tell about data. Read the data first. | Paste the literal error in Phase 1. If the message names a file:line, look at that file:line before declaring the message is wrong. Most "misleading" errors are accurate; the engineer was holding a wrong mental model of which code runs first. |144145## Evidence Requirements146147Every phase has a gate. If the gate's artifact does not exist, that phase has not148been completed.149150| Checkpoint | Required artifact | What "no evidence" looks like |151|---|---|---|152| End of Phase 1 | Literal error text + reproducer command + relevant commit hashes pasted in a `Phase 1: Gather` block | "I read through the code and I'm pretty sure it's in the auth module." |153| End of Phase 2 | One sentence in form `The bug occurs because [X] causes [Y] when [Z]` | "It's probably a race condition somewhere in the request lifecycle." |154| End of Phase 3 | Captured output from a deliberate test of the hypothesis (instrumentation OR experiment), plus a confirm/refute verdict | "Yeah I tried a thing and it seemed to work." |155| End of Phase 4 | Failing test (red), passing test after fix (green), full suite (green), original reproducer (fixed) — all four pasted | "Tests pass on my machine." |156157If you can't paste it, you haven't done it. Stop.158159## Red Flags160161Concrete observations that mean STOP and reassess.162163- You've changed the same line three or more different ways in the last hour. You164 don't have a working hypothesis; you're guessing.165- You added a `try/catch`, `if err == nil`, or test-skip whose justification is166 "to make the test pass." That's masking, not fixing.167- The hypothesis sentence is missing the `when [Z]` clause. You don't know the168 trigger condition. The fix will be partial.169- Three consecutive fix attempts have failed. The bug is architectural, not local.170 Escalate or rescope.171- You're about to ship a fix you cannot explain to the next reviewer in one172 sentence. The reviewer won't accept it; you shouldn't either.173- The failing test you wrote in Phase 4 doesn't actually fail on `main` without174 the fix. It tests something tangential. Rewrite it.175176## References177178- John Allspaw & Richard Cook, *How Complex Systems Fail* (Cognitive Technologies179 Laboratory, 1998) — point #5 ("Complex systems run in degraded mode") and point180 #14 ("Change introduces new forms of failure"). Use these to resist the "it181 worked before the deploy" reflex; the post-deploy failure is often a latent182 problem made visible, not the deploy itself.183- *Site Reliability Engineering*, Beyer et al. (Google, O'Reilly 2016), Chapter 12184 "Effective Troubleshooting" — defines the diagnose-test-fix loop this skill's185 Phases 2-4 implement, and explicitly warns against the "I know what's wrong"186 pattern handled in the Rationalizations table.
duthaho