Failure Forensics

Failure Forensics

Establish the causal chain before changing the system. Preserve the user's boundary: a request to diagnose or review does not authorize a fix.

Build the case

1. Define the failure precisely.

   • Record the expected behavior, first observable deviation, exact command or action, working directory, revision, and relevant environment.
   • Capture the complete first failure. Treat later errors as possible fallout until proven otherwise.
   • Redact secrets and personal data from commands, logs, and reports.

2. Establish a baseline.

   • Reproduce the failure without changing the system first.
   • For an intermittent failure, run a bounded sample of 3-10 identical trials and record the rate, timing, and conditions. Do not loop indefinitely.
   • Identify the earliest layer that returns an anomalous value: input, configuration, dependency, process, network, persistence, or presentation.

3. Keep a hypothesis ledger.

   • Create 2-5 competing explanations that make different predictions.
   • For each one, record: supporting evidence, contradicting evidence, predicted observation, cheapest safe test, and result.
   • Rank tests by information gained per unit of time and risk, not by how familiar the suspected cause feels.

4. Run the smallest discriminating experiment.

   • Change one dimension at a time while holding the reproduction path constant.
   • Prefer read-only inspection, binary isolation, known-good comparisons, dependency-boundary probes, and temporary local overrides.
   • Avoid production mutations, destructive cleanup, broad upgrades, or deleting state merely to see whether the problem disappears.

5. Apply the causality bar.

   • Call a cause confirmed only when changing it produces the predicted result and either reversing the change restores the failure or independent evidence corroborates the mechanism.
   • Call it probable when multiple independent observations support it but a reversible intervention is impractical.
   • Call it plausible when it remains an untested explanation.
   • If reproduction is impossible, state that clearly and propose the smallest instrumentation needed for the next occurrence.

Deliver the verdict

Report in this order:

1. Failure - the precise break and reproduction status.
2. Diagnosis - the earliest faulty boundary and confidence level.
3. Evidence - observations and experiments that establish the chain.
4. Ruled out - credible alternatives eliminated by evidence.
5. Next move - the smallest safe correction or next probe.

If the user authorized a fix, implement the smallest correction that addresses the confirmed mechanism, add a targeted regression check, then run the relevant broader checks. If the user asked only for diagnosis, stop before editing and describe the proposed correction instead.