Tracing Root Causes

Tracing root causes

Explain outcomes through evidence, not narrative, and fix causes rather than symptoms. The failure mode this prevents: jumping from symptom to favorite explanation, collecting only confirming evidence, then patching where the error surfaced.

The iron law

NO FIX WITHOUT A ROOT CAUSE FIRST

A fix proposed before the cause is known is a guess wearing a diff. This holds hardest exactly where it feels most expensive — during an outage, at hour four, with a senior engineer already typing.

Two things it does not forbid:

• Mitigation under a live incident. Stopping the bleeding is legitimate — a rollback, a circuit breaker, a feature flag. It is named as mitigation, kept reversible, and it does not end the investigation or get recorded as the cause.
• Doing what you were asked. If the owner has heard the concern and still wants their fix, write it, state the assumption it rests on and what would falsify it, and move on. State a disagreement once, plainly — never twice, and never as silent compliance.

The phases

Run them in order. A phase skipped is a phase paid for later.

1. Reproduce

• Read the error completely — full message, full stack, line numbers, codes. It frequently names the answer.
• Restate exactly what was observed — which artifact, what behavior, when. If you catch yourself rewriting the observation to fit a theory, stop.
• Trigger it reliably. What are the exact steps? Every time, or one run in five? Not reproducible is a finding, not a dead end: it points at timing, environment, or shared state.
• Check what changed — recent commits, dependency bumps, config and environment differences between where it fails and where it doesn't.

2. Isolate

Narrow the failure to one component before explaining it.

• Instrument the boundaries. In a multi-component system (CI → build → sign, API → service → database), log what enters and what leaves each boundary, run once, and read which hop breaks. Evidence about where beats speculation about why.
• Compare against something that works. Find the nearest working case and list every difference, however small. "That can't matter" is a hypothesis, not an observation.
• Check what it depends on — the services, config, environment, and assumptions the broken path needs to work at all. A missing or changed dependency looks exactly like broken logic from the inside.
• Trace backward to the origin — see references/techniques.md for backward tracing, instrumenting a dangerous operation, and bisecting a test suite to find which test dirties the tree.

3. Hypothesize

• Compete the explanations. Hold ≥2 from deliberately different frames: code path, config/environment, measurement artifact, external dependency, timing, shared state. "The measurement is wrong" is always a candidate.
• Rank evidence by strength. Strongest to weakest: controlled repro / discriminating experiment → primary artifact with tight provenance (timestamped logs, git history, file:line behavior) → independent sources converging → single-source inference that fits → circumstantial (naming, temporal proximity, stack position, "this same fix worked on another service") → intuition, seniority, confidence. When tiers conflict the higher tier wins; never treat support as flat.
• Name what you don't understand. "I don't know how X works yet" is a finding, not a confession — say it, then go read X completely. A hypothesis resting on an unread component is a guess about that component, and a confident verdict about code nobody has read is the most expensive kind of wrong.

4. Disconfirm

• Ask the two questions of each serious hypothesis: "what should we observe if this were true — do we?" and "what observation would be hard to explain if this were true?" A hypothesis that survives only because nobody looked for counter-evidence keeps LOW confidence.
• Run a rebuttal round. Let the strongest alternative challenge the leader with its best contrary evidence before you conclude. Down-rank explanations that need fresh unverified assumptions to survive, and don't merge hypotheses that merely sound alike (fake convergence).
• Test minimally. One variable, the smallest change that discriminates. A bigger timeout after a failed timeout is not a new hypothesis — it is the same one, retried louder.
• Name the critical unknown — the single missing fact behind most of the remaining uncertainty — and the ONE probe that best separates the top hypotheses. Prefer probes that discriminate over probes that gather more of the same support, and pick the cheapest one that still discriminates — under time pressure that is often minutes of reading the actual implementation, not the full investigation someone is refusing to fund.

5. Fix

• Write the failing reproduction first. Simplest case that fails; a test if there's a framework, a script if there isn't. It proves the cause and proves the fix.
• Fix at the source, not where the error surfaced. One change, one concern — no bundled refactor, no "while I'm here".
• Then make it impossible. Add validation at the layers the bad value crossed — entry, business logic, environment guard, instrumentation (references/techniques.md — defense in depth, and condition-based waiting when the cause was timing).
• Verify, then hand off. Confirm the reproduction now passes and nothing else broke — then stop; see Where this ends.

Three strikes → question the architecture

Count fix attempts. At three failures, the problem is no longer the hypothesis:

• Each fix reveals a new problem somewhere else.
• Each fix needs "just a bit of refactoring" to land.
• The same symptom keeps returning in a new costume.

Stop. Say plainly that this looks structural, name the pattern you think is wrong, and get a decision before attempt four. This is not a failed hypothesis — it is a wrong architecture, and more fixes make it worse.

When the investigation stalls

Reachable evidence is spent and the phase-4 critical unknown is a fact this seat cannot produce — access it lacks, knowledge a human holds, a decision that is the owner's. That is a stall, and its one honest exit is to ask. Never a silent guess, never a silent stop: a deadline bounds which probes you can afford, not whether you report.

• Earn the ask. An affordable discriminating probe still on the table means run it, not ask — minutes of reading the implementation outrun waiting for an answer. Escalate when the next probe is one you cannot run.
• Name the recipient — the owner, or the parent that dispatched you. "Owner: someone else", written inside a document you then submit, is an assignment, not a question.
• Use a channel that waits — a blocking ask upward, an interactive question in a live session. A file read after the lane closes is a record, not a channel.
• Carry the evidence; it is the price of asking. Hypotheses tried and the observation that disconfirmed each, the exact point of the stall, the ONE fact you need, and the probe that runs the moment it comes back. A bare "I'm stuck" hands the answerer your investigation to redo.

When the cause really is environmental

Sometimes the honest verdict is timing, an external dependency, or a platform difference, with no defect to fix. That verdict is earned, not reached by running out of patience — most "no root cause" is incomplete investigation. When the trace genuinely lands there:

1. Record what was ruled out, and how. The next person starts from your shortlist instead of from zero.
2. Handle it deliberately — a bounded retry, a timeout, a clear error message. Deliberate handling is a design decision with a rationale; a swallowed error is not.
3. Add the monitoring that would make the next occurrence legible, because there will be a next occurrence.

Rationalizations

Redirection from a human — "is that not happening?", "stop guessing", "we're stuck?" — means the same thing: return to phase 1.

Output shape

Ranked hypothesis table (confidence + evidence strength + why it survives), evidence for AND against each, current best explanation (explicitly provisional when evidence is incomplete), critical unknown, discriminating probe. When a fix follows, the reproduction, the change, and what it means if the change didn't help.

Blocked by missing evidence? The ranked shortlist plus the probe IS the deliverable, not a failure — and it is delivered to someone, not filed; see When the investigation stalls.

Where this ends

This skill investigates and fixes. It does not certify.

• Evidence lands in the lane's PROGRESS.md — the reproduction, the ranked hypotheses, the probe results, the change. Not a scratch debugging log, not the chat alone.
• work-verify owns the completion gate. Hand off there. "The test passes now" is a result to be verified, never a verdict.
• work-handoff closes the lane. This skill never declares work done, fixed, or shippable on its own authority.

Judgment notes

• Investigation only? Stop after phase 4 and deliver the trace. Asked to fix as well? Finish the trace first, then fix — the phases are the order, not a menu.
• Framing pressure ("confirm it was the deploy so we can roll back") is a request for a conclusion, not for analysis. Give the analysis; if it supports the framing, say so with the evidence.
• This skill supersedes superpowers:systematic-debugging, whose machinery it absorbed. Superpowers remains installed as the fallback where this library is not.