Systematic Debugging

Purpose

Find the root cause of any bug before attempting a fix. Random fixes waste time and create new bugs. Symptom fixes are failure.

Core principle: ALWAYS find root cause before attempting fixes.

When to Use

Use for ANY technical issue:

• Test failures
• Bugs in production
• Unexpected behavior
• Performance problems
• Build failures
• Integration issues

Use this especially when:

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• You've already tried multiple fixes
• Previous fix didn't work
• You don't fully understand the issue

Do not skip when:

• Issue seems simple (simple bugs have root causes too)
• You're in a hurry (rushing guarantees rework)
• Someone wants it fixed immediately (systematic is faster than thrashing)

Inputs

• Bug description or error message.
• Stack trace or reproduction steps.
• Environment context (OS, versions, recent changes).
• Access to the codebase and test suite.

Workflow

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

Before attempting ANY fix:

1. Read error messages carefully — Don't skip past errors or warnings. Read stack traces completely. Note line numbers, file paths, error codes.

2. Reproduce consistently — Can you trigger it reliably? What are the exact steps? If not reproducible: gather more data, don't guess.

3. Check recent changes — What changed that could cause this? Git diff, recent commits, new dependencies, config changes, environmental differences.

4. Gather evidence in multi-component systems — When the system has multiple components (CI → build → signing, API → service → database):

   For EACH component boundary:
     - Log what data enters the component
     - Log what data exits the component
     - Verify environment/config propagation
     - Check state at each layer
   
   Run once to gather evidence showing WHERE it breaks.
   Then analyze to identify the failing component.
   Then investigate that specific component.
   

5. Trace data flow — Where does the bad value originate? What called this with the bad value? Keep tracing up until you find the source. Fix at source, not at symptom. See references/root-cause-tracing.md for the complete backward tracing technique.

Phase 2: Pattern Analysis

1. Find working examples — Locate similar working code in the same codebase.
2. Compare against references — Read reference implementations completely. Don't skim.
3. Identify differences — List every difference between working and broken, however small.
4. Understand dependencies — What other components, settings, or config does this need?

Phase 3: Hypothesis and Testing

1. Form a single hypothesis — "I think X is the root cause because Y." Write it down. Be specific.
2. Test minimally — Make the smallest possible change to test the hypothesis. One variable at a time.
3. Verify before continuing — Did it work? Yes → Phase 4. No → form a new hypothesis. Do not add more fixes on top.
4. When you don't know — Say so. Ask for help. Research more. Do not pretend to know.

Phase 4: Implementation

1. Create a failing test case — Simplest possible reproduction. Automated test if possible. Must exist before fixing.

2. Implement a single fix — Address the root cause identified. One change at a time. No "while I'm here" improvements.

3. Verify the fix — Test passes? No other tests broken? Issue actually resolved?

4. If fix doesn't work — STOP. Count how many fixes you've tried.

   • If fewer than 3: return to Phase 1, re-analyze with new information.
   • If 3 or more: question the architecture (see below).

5. If 3+ fixes failed: question the architecture — Each fix revealing new shared state or coupling elsewhere is a sign of an architectural problem, not a hypothesis failure. Stop and discuss with the team before attempting another fix.

Output

Root cause: <what caused the bug — specific, not vague>
Evidence: <what you observed that confirmed the hypothesis>
Fix: <what was changed and where>
Test: <test added or updated>
Verification: <how fix was confirmed>

Verification

• Root cause found (not just symptom location)
• Single hypothesis tested at a time
• Minimal fix applied — no bundled changes
• Test added or updated
• If 3+ fixes failed: architectural discussion initiated, not another fix attempt

Failure Modes

Red flags — STOP and return to Phase 1:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• Adding multiple changes at once
• "Skip the test, I'll manually verify"
• "It's probably X" without evidence
• "I don't fully understand but this might work"
• "Here are the main problems: [lists fixes without investigation]"
• Proposing solutions before tracing data flow
• "One more fix attempt" (when already tried 2+)
• Each fix reveals a new problem in a different place

Common rationalizations:

Supporting References

Techniques available in references/:

• root-cause-tracing.md — Trace bugs backward through the call stack to find the original trigger.
• defense-in-depth.md — Add validation at multiple layers after finding the root cause.
• condition-based-waiting.md — Replace arbitrary timeouts with condition polling to fix flaky tests.
• find-polluter.sh — Bisection script to identify which test creates unwanted files or state.

Example implementation in examples/:

• condition-based-waiting-example.ts — Complete TypeScript implementation of condition-based waiting utilities.