# Root Cause Investigation

> Investigate a defect, an outage or a surprising behaviour down to the actual cause — reproduce first, shrink the reproduction, separate correlation from cause, and stop at the real mechanism rather than the first plausible story. Use for any bug report, incident, flaky test, or "this number looks wrong", and before writing a fix.

- Skill: `chinthakat/root-cause-investigation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add chinthakat/root-cause-investigation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/chinthakat/root-cause-investigation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: chinthakat (https://skillmd.com/u/chinthakat)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/chinthakat/root-cause-investigation

---


# Root cause investigation

The failure mode of debugging is not being unable to find the cause. It is
finding *a* cause — plausible, adjacent, and wrong — shipping a fix, and having
the bug come back with a different face.

## 1. Reproduce before you fix

If you cannot make it happen on demand, you cannot know you fixed it. You can
only know it has not happened again yet, which is what "it's intermittent"
means.

Reproducing is the work. Budget most of the time for it, and be explicit about
what you have:

| State | What you can honestly claim |
|---|---|
| Reproduced on demand | You can verify a fix |
| Reproduced sometimes | You can verify a fix statistically, slowly |
| Never reproduced | You are guessing. Say so in the commit message |

If it cannot be reproduced, the first deliverable is not a fix — it is better
instrumentation, shipped so the next occurrence is diagnosable.

## 2. Write down what you know and what you assume

Two columns, literally:

```
KNOWN                                  ASSUMED
Error at 09:14:22 UTC, 3 occurrences   It started after Tuesday's deploy
User ids: u_44, u_91, u_102            All three are on the mobile app
Stack: parse() line 88                 The input is malformed JSON
```

The assumed column is where the investigation goes wrong, because assumptions
get treated as facts within about ten minutes. Move each one across only when
you have checked it, and check the cheap ones first — "all three are on mobile"
is one query and it either narrows the search enormously or kills a theory.

## 3. Read the whole error

Read it slowly, all of it, and out loud if necessary:

- The message, exactly. Not what you expect it to say.
- **The stack below the first frame.** The first frame is where it surfaced;
  the cause is usually three frames down.
- The `caused by` chain to the bottom. The bottom is the real one.
- The timestamp, and what else happened at that timestamp.
- Whether this is the first occurrence or the ten-thousandth. Log volume is
  evidence.

More investigations are ended by carefully re-reading the error than by any
other technique, and it is the step most often skipped in favour of a theory.

## 4. Shrink the reproduction

Once it reproduces, make it smaller. Each dimension you can eliminate removes a
whole class of cause:

- **Time** — `git bisect` between a known-good and known-bad commit. This is the
  highest-value tool available and is consistently under-used. Automate the test
  and let it run.
- **Input** — halve the data until it stops failing. One row that fails is a
  different investigation from ten thousand that fail.
- **Environment** — does it fail locally? In CI? Only in production? The
  difference between two environments *is* the answer, and enumerating the
  differences is faster than guessing.
- **Configuration** — flags, feature toggles, tenant settings.
- **Identity** — one user, one tenant, one role, or everyone?

A minimal reproduction is also the test you are about to write.

## 5. Correlation is not cause

"It started after the deploy" is the commonest false lead, and it is seductive
because it is usually *partly* true — the deploy exposed something that was
already wrong, or changed timing enough to surface a latent race.

Before accepting a correlation:

- Does the mechanism connect? Name the line of the change that causes the
  symptom. If you cannot, you have a coincidence.
- Did it really start then, or did you start *looking* then? Check the logs
  before the deploy for the same signature.
- Was anything else deployed, migrated, scaled, expired or rotated at that time?
  Certificates, tokens, cron jobs and quota resets all have anniversaries.

## 6. Change one thing at a time

Under pressure the instinct is to change five things and see if it goes away. It
often does, and then nobody knows which one mattered, so all five stay, and one
of them is a new bug.

One change, observe, revert if it did nothing. Keep a log of what you tried and
what happened — in an incident it becomes the timeline, and in a long
investigation it stops you retrying the same thing on day two.

## 7. Stop at the mechanism, not at the first plausible story

Keep asking "why does that happen?" until the answer is something you can point
at in code, configuration or data.

```
The report is empty.
  Why? The query returns no rows.
    Why? The tenant filter doesn't match.
      Why? The tenant id is null on the request.
        Why? The token has no tenant claim.
          Why? Tokens minted by the CLI path skip the claim enricher.   <- mechanism
```

Stopping at "the tenant id is null" produces a null check, which makes the
symptom disappear and leaves every CLI-minted token quietly unscoped. The
difference between the fourth and fifth line is the difference between a patch
and a fix.

Then ask the last question: **why did nothing catch this?** The missing test,
the missing alert or the missing type is usually a more valuable fix than the
bug itself.

## 8. The fix carries a test that failed first

Write the test, watch it fail for the right reason — the reason matters; a test
that fails because of a typo proves nothing — then fix, then watch it pass.

If the code has no harness at that layer, say so in the commit rather than
pretending. An honest "not covered, verified manually by X" is worth more than a
test that exercises a mock of the thing that broke.

## 9. Look for siblings

A bug is rarely unique. The same mistake is usually elsewhere:

- Grep for the pattern, not the symptom. If a token path skipped an enricher,
  find every path that mints a token.
- Check the sibling call sites of whatever was wrong.
- Check whether the same class of input breaks a different feature.

Fixing the reported one and leaving four is how the same incident recurs with a
different title.

## 10. Write it down, even unfinished

A stalled investigation still produces value if it is recorded: what was
observed, what was ruled out and how, what remains suspected. The next person —
often you — starts from the frontier instead of the beginning.

Record it the way `issue-ledger` describes: entity ids, the environment, the
date, and observation separated from inference.

## Checklist

- [ ] Reproduced, or the honest state of reproduction stated
- [ ] Known and assumed written separately; assumptions checked before use
- [ ] The whole error read, including the bottom of the `caused by` chain
- [ ] Reproduction shrunk along time, input, environment, config and identity
- [ ] Bisect used where there is a known-good commit
- [ ] Every correlation either given a mechanism or discarded
- [ ] One change at a time, with a log of what was tried
- [ ] Asked "why" down to something pointable-at in code, config or data
- [ ] Asked why nothing caught it, and fixed that too
- [ ] Test written that failed first, for the right reason
- [ ] Siblings searched for by pattern
- [ ] Findings recorded even if unresolved