Debugging By Evidence

Debugging by Evidence

Core principle. Test the claim needed for the decision, not every unknown in the incident.

• A hypothesis chooses an experiment; it is not a confirmed cause. Reports, logs and code can supply falsifiable hypotheses before a local reproduction.
• Use complementary layers of evidence. An incident record shows occurrence; a controlled experiment can show defective handling without identifying the original trigger.
• You opened this in the middle of something. This is how to do that work, not a replacement for it. Name what you were doing before you start, and return to it when this is done.

Choose a signal for the question

• Start with the reported failure and available evidence. A local passing run does not erase a recorded incident.
• Choose reproduction, replay, controlled fault injection or tracing by what each can establish. They are alternatives, not a mandatory ladder.
• Keep the handling under test real. Inject a justified dependency outcome or schedule, then assert the caller-visible contract.
• Record the command, conditions and result. Label injected conditions; do not report them as observed production events.

What you may write, and when

• Before a supported repair, writes are limited to authorised diagnostic tests and instrumentation. Controlled changes to inputs, dependency outcomes or scheduling are experiments, not production fixes.
• Preserve the baseline. Change the condition under investigation, not the handling whose correctness you are testing. Isolate substitutions and retain the original evidence.
• The licence stops at the workspace boundary. No commit, branch move, remote action or deployment. Remove temporary instrumentation; retain useful regression tests.

Establish before the first run

• Never ask what the environment answers. Three things settle the setup, and the environment already holds two of them.

  • The symptom in the user's own words, quoted.
  • The command the project already declares for running that surface.
  • Whether the failure is reported as constant or occasional, and which conditions or observations are available.

• Distinguish a reported rate from a measured one. Measure frequency only when it informs the decision; label unknown frequency rather than requiring repeated failures before investigating.

• Say the budget out loud before the first probe. How far you intend to go, in probes or in minutes, stated where I can see it. A bounded investigation I can interrupt is worth more than an unbounded one that arrives finished.

• Each next probe needs a decision it can inform. Stop, change the seam or hand back when another run cannot answer the remaining question within scope.

Which rules to read

One rule per row. Match the left column against the symptom or the state you are in.

• The match sets where to start. Evidence determines what is justified next; the states are not a sequence every investigation must traverse.
• Read a rule when its decision is needed. Stopping and authority limits apply from every state.
• Where two rows both look like the symptom, read both. Under-reading costs a whole loop. Over-reading costs one file.
• Read every row, then act on the matches, hardest to undo first. Reading a row costs nothing; the row you skipped is where the coverage went.

Discriminators.

• Signal against minimising. Signal chooses a runnable observation or experiment. Minimising removes irrelevant setup once a useful signal exists.
• Hypotheses against probing. Name the prediction before the probe. Independent questions may proceed at different layers; keep their evidence separate.
• Source against seam. Source decides where a supported repair belongs. Seam decides which contract its test protects.

Default stance.

• Use the cheapest adequate evidence, not the cheapest check regardless of what it proves.
• Keep independent fronts bounded. Source analysis, a handling test and environment observation need not wait for each other when their inputs are available. Do not mix overlapping experimental edits.
• Do not assert more than was tested. A conditional handling defect can justify a scoped repair without establishing what triggered a past incident.

Say what the evidence permits

Report state when it changes, with the established claim and remaining gap.

• Do not minimise for its own sake. Stop reducing when the experiment is understandable and sufficient for the decision.
• A gap blocks only claims and actions that depend on it. Investigate another independent layer when useful; do not keep chasing the same unavailable observation.
• A passing run has limited scope. No event observed is not proof that an intermittent event cannot occur.

Match the conclusion to the evidence

• Occurrence: a reliable incident record or reproduction shows the failure happened in those conditions.
• Handling: a controlled test shows how real code reacts to a specified condition. A dependency outcome must be compatible with its contract; an invented behaviour is not a reproduction.
• Historical cause: identifying the trigger of the original incident needs evidence of that trigger, not just a test that forces it.
• Repair: removing a defective dependency or branch needs a failing contract check and compatibility checks. It need not require a trigger-rate estimate.
• Tuning: retries, backoff or timeout changes need evidence relevant to recovery, timing, load and side effects. A forced error alone does not justify the chosen values.

For each causal claim, cite the code or observed event, the discriminating result and what remains unknown. A plausible alternative need not be eliminated if it does not change the justified repair; record that boundary.

When to stop instead of trying again

• Name what another run could change. Repetition can estimate variability under stated conditions, but is not automatically progress toward the cause.
• Treat new regressions as evidence against the repair, not automatic proof that the architecture is wrong.
• Stop when the scoped decision is supported, an essential observation is inaccessible, the investigation limit is reached, or the next step needs new authority. Report unresolved questions without blocking unrelated supported work.

Output contract

Symptom      as reported, with the available incident evidence
State        the established claim and the next justified action
Experiments  conditions, real code exercised, observed or injected inputs, results
Mechanism    supported causal links at file:line, including conditional claims
Not shown    historical trigger, frequency or compatibility still unverified
Fix          the scoped change and affected callers/contracts
Proof        the failing contract check before and after, relevant adjacent checks,
             and removal of temporary instrumentation

• Report what you observed, not what you avoided.
• Never open with "fixed". The reader needs the chain first. A fix stated before its cause reads as a guess that happened to work, and sometimes it is one.

Do not skip this when

• The cause seems obvious. That is the anchor this skill exists to break.
• Someone already reported the failure. Use their evidence; do not demand a fresh local occurrence before reasoning about it.
• The fix is one line. A one-line fix to the wrong line is still wrong.
• You are in a hurry. Guessing is what produces the second and third attempt.

Routing

• The table above selects the rule and its order. Read a selected rule in full, interpret it yourself, and say which one you opened in one line.
• Rival explanations MAY be tested in parallel. Ranking them and judging what a result means is not delegated.
• A direct instruction from the user outranks anything here.
• Once a fix exists it stops being a symptom and becomes a change. Hand it to a review of the diff rather than judging it here.