Root Cause Investigation

Root cause investigation

The failure mode of debugging is not being unable to find the cause. It is finding a cause — plausible, adjacent, and wrong — shipping a fix, and having the bug come back with a different face.

1. Reproduce before you fix

If you cannot make it happen on demand, you cannot know you fixed it. You can only know it has not happened again yet, which is what "it's intermittent" means.

Reproducing is the work. Budget most of the time for it, and be explicit about what you have:

If it cannot be reproduced, the first deliverable is not a fix — it is better instrumentation, shipped so the next occurrence is diagnosable.

2. Write down what you know and what you assume

Two columns, literally:

KNOWN                                  ASSUMED
Error at 09:14:22 UTC, 3 occurrences   It started after Tuesday's deploy
User ids: u_44, u_91, u_102            All three are on the mobile app
Stack: parse() line 88                 The input is malformed JSON

The assumed column is where the investigation goes wrong, because assumptions get treated as facts within about ten minutes. Move each one across only when you have checked it, and check the cheap ones first — "all three are on mobile" is one query and it either narrows the search enormously or kills a theory.

3. Read the whole error

Read it slowly, all of it, and out loud if necessary:

• The message, exactly. Not what you expect it to say.
• The stack below the first frame. The first frame is where it surfaced; the cause is usually three frames down.
• The caused by chain to the bottom. The bottom is the real one.
• The timestamp, and what else happened at that timestamp.
• Whether this is the first occurrence or the ten-thousandth. Log volume is evidence.

More investigations are ended by carefully re-reading the error than by any other technique, and it is the step most often skipped in favour of a theory.

4. Shrink the reproduction

Once it reproduces, make it smaller. Each dimension you can eliminate removes a whole class of cause:

• Time — git bisect between a known-good and known-bad commit. This is the highest-value tool available and is consistently under-used. Automate the test and let it run.
• Input — halve the data until it stops failing. One row that fails is a different investigation from ten thousand that fail.
• Environment — does it fail locally? In CI? Only in production? The difference between two environments is the answer, and enumerating the differences is faster than guessing.
• Configuration — flags, feature toggles, tenant settings.
• Identity — one user, one tenant, one role, or everyone?

A minimal reproduction is also the test you are about to write.

5. Correlation is not cause

"It started after the deploy" is the commonest false lead, and it is seductive because it is usually partly true — the deploy exposed something that was already wrong, or changed timing enough to surface a latent race.

Before accepting a correlation:

• Does the mechanism connect? Name the line of the change that causes the symptom. If you cannot, you have a coincidence.
• Did it really start then, or did you start looking then? Check the logs before the deploy for the same signature.
• Was anything else deployed, migrated, scaled, expired or rotated at that time? Certificates, tokens, cron jobs and quota resets all have anniversaries.

6. Change one thing at a time

Under pressure the instinct is to change five things and see if it goes away. It often does, and then nobody knows which one mattered, so all five stay, and one of them is a new bug.

One change, observe, revert if it did nothing. Keep a log of what you tried and what happened — in an incident it becomes the timeline, and in a long investigation it stops you retrying the same thing on day two.

7. Stop at the mechanism, not at the first plausible story

Keep asking "why does that happen?" until the answer is something you can point at in code, configuration or data.

The report is empty.
  Why? The query returns no rows.
    Why? The tenant filter doesn't match.
      Why? The tenant id is null on the request.
        Why? The token has no tenant claim.
          Why? Tokens minted by the CLI path skip the claim enricher.   <- mechanism

Stopping at "the tenant id is null" produces a null check, which makes the symptom disappear and leaves every CLI-minted token quietly unscoped. The difference between the fourth and fifth line is the difference between a patch and a fix.

Then ask the last question: why did nothing catch this? The missing test, the missing alert or the missing type is usually a more valuable fix than the bug itself.

8. The fix carries a test that failed first

Write the test, watch it fail for the right reason — the reason matters; a test that fails because of a typo proves nothing — then fix, then watch it pass.

If the code has no harness at that layer, say so in the commit rather than pretending. An honest "not covered, verified manually by X" is worth more than a test that exercises a mock of the thing that broke.

9. Look for siblings

A bug is rarely unique. The same mistake is usually elsewhere:

• Grep for the pattern, not the symptom. If a token path skipped an enricher, find every path that mints a token.
• Check the sibling call sites of whatever was wrong.
• Check whether the same class of input breaks a different feature.

Fixing the reported one and leaving four is how the same incident recurs with a different title.

10. Write it down, even unfinished

A stalled investigation still produces value if it is recorded: what was observed, what was ruled out and how, what remains suspected. The next person — often you — starts from the frontier instead of the beginning.

Record it the way issue-ledger describes: entity ids, the environment, the date, and observation separated from inference.

Checklist

• Reproduced, or the honest state of reproduction stated
• Known and assumed written separately; assumptions checked before use
• The whole error read, including the bottom of the caused by chain
• Reproduction shrunk along time, input, environment, config and identity
• Bisect used where there is a known-good commit
• Every correlation either given a mechanism or discarded
• One change at a time, with a log of what was tried
• Asked "why" down to something pointable-at in code, config or data
• Asked why nothing caught it, and fixed that too
• Test written that failed first, for the right reason
• Siblings searched for by pattern
• Findings recorded even if unresolved