Debugging Helpers

Systematic Debugging

Find root cause before attempting fixes. Symptom fixes mask underlying issues and waste time.

The Four Phases

Complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

Before attempting any fix:

1. Read error messages carefully

   • Read stack traces completely — they often contain the exact solution
   • Note line numbers, file paths, error codes

2. Reproduce consistently

   • Identify exact steps to trigger reliably
   • If not reproducible, gather more data before forming hypotheses

3. Check recent changes

   • Git diff, recent commits, new dependencies, config changes
   • Environmental differences

4. Gather evidence in multi-component systems

   When a system has multiple components (CI → build → signing, API → service → database), add diagnostic instrumentation before proposing fixes:

   For each component boundary:
     - Log what data enters and exits the component
     - Verify environment/config propagation
     - Check state at each layer
   
   Run once to gather evidence showing WHERE it breaks
   Then investigate that specific component
   

   Example (multi-layer system):

   # Layer 1: Workflow
   echo "=== Secrets available: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
   
   # Layer 2: Build script
   env | grep IDENTITY || echo "IDENTITY not in environment"
   
   # Layer 3: Signing
   security find-identity -v
   
   # Layer 4: Actual signing
   codesign --sign "$IDENTITY" --verbose=4 "$APP"
   

5. Trace data flow

   See root-cause-tracing.md for the complete backward tracing technique.

   Quick version:

   • Where does the bad value originate?
   • What called this with the bad value?
   • Keep tracing up until you find the source
   • Fix at source, not at symptom

Phase 2: Pattern Analysis

1. Find working examples — locate similar working code in the same codebase
2. Compare against references — read reference implementations completely before applying
3. Identify differences — list every difference between working and broken, however small
4. Understand dependencies — what components, settings, config, or assumptions are involved?

Phase 3: Hypothesis and Testing

1. Form a single hypothesis — state clearly: "I think X is the root cause because Y"
2. Test minimally — make the smallest possible change to test the hypothesis; one variable at a time
3. Verify before continuing — if it worked, proceed to Phase 4; if not, form a new hypothesis (avoid stacking fixes)

Phase 4: Implementation

1. Create a failing test case — simplest possible reproduction, automated if feasible

2. Implement a single fix — address root cause only; no "while I'm here" improvements

3. Verify the fix — test passes, no other tests broken, issue actually resolved

4. If the fix fails — count attempts:

   • < 3 attempts: return to Phase 1, re-analyze with new information
   • ≥ 3 attempts: stop and question the architecture (see below)

5. If 3+ fixes failed: question architecture

   Pattern indicating architectural problems:

   • Each fix reveals new shared state / coupling / problem in a different place
   • Fixes require massive refactoring to implement
   • Each fix creates new symptoms elsewhere

   Stop and question fundamentals:

   • Is this pattern fundamentally sound?
   • Should we refactor architecture vs. continue fixing symptoms?

   Discuss with the human before attempting more fixes.

Signs You Should Return to Phase 1

If any of these apply, pause and re-investigate before continuing:

• Attempting a fix without understanding the root cause
• Skipping reproduction or test verification
• Stacking multiple speculative changes
• On the 3rd+ fix attempt without new evidence
• Proposing solutions before tracing data flow

Supporting Techniques

Available in this directory:

• root-cause-tracing.md — trace bugs backward through call stack to find original trigger
• defense-in-depth.md — add validation at multiple layers after finding root cause
• condition-based-waiting.md — replace arbitrary timeouts with condition polling

Related skills:

• tdd — for creating failing test cases (Phase 4)
• dev-testing — verify fix worked before claiming success

Quick Reference