Systematic Debugging

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find root cause before attempting fixes.

When to Use

Use for ANY technical issue: test failures, unexpected behavior, pipeline errors, build failures, Galaxy workflow errors, notebook exceptions, environment issues.

Use ESPECIALLY when:

• "Just one quick fix" seems obvious
• You've already tried multiple fixes
• Previous fix didn't work
• You don't fully understand the issue

Supporting Files

• root-cause-tracing.md - Trace bugs backward through call chain to find the original trigger. Instrumentation techniques, stack trace analysis.
• defense-in-depth.md - Add validation at multiple layers after finding root cause. Entry point, business logic, environment guards, debug logging.

The Four Phases

Complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read error messages carefully

   • Don't skip past errors or warnings — they often contain the solution
   • Read stack traces completely
   • Note line numbers, file paths, error codes

2. Reproduce consistently

   • Can you trigger it reliably? What are the exact steps?
   • If not reproducible, gather more data — don't guess

3. Check recent changes

   • Git diff, recent commits, new dependencies
   • Config changes, environmental differences

4. Gather evidence in multi-component systems

   • For pipelines (Galaxy workflow → tool → data), log what enters and exits each component
   • Run once with diagnostics to see WHERE it breaks
   • Then investigate that specific component

5. Trace data flow

   • Where does the bad value originate? (See root-cause-tracing.md)
   • Keep tracing up the call chain until you find the source
   • Fix at source, not at symptom

Phase 2: Pattern Analysis

1. Find working examples — similar working code in same codebase
2. Compare against references — read reference implementation completely, don't skim
3. Identify differences — list every difference, however small
4. Understand dependencies — settings, config, environment, assumptions

Phase 3: Hypothesis and Testing

1. Form single hypothesis — "I think X is the root cause because Y"
2. Test minimally — smallest possible change, one variable at a time
3. Verify — did it work? If not, form NEW hypothesis. Don't pile fixes on top.
4. When you don't know — say so. Don't pretend. Research more.

Phase 4: Implementation

1. Create failing test/reproduction — simplest possible, automated if possible
2. Implement single fix — address root cause, ONE change, no "while I'm here" improvements
3. Verify fix — test passes? No other tests broken? Issue resolved?
4. If fix doesn't work:
   • Count fixes attempted
   • If < 3: return to Phase 1, re-analyze with new information
   • If >= 3: STOP — question the architecture (see below)

When 3+ Fixes Fail

Pattern indicating architectural problem:

• Each fix reveals new issues in different places
• Fixes require "massive refactoring"
• Each fix creates new symptoms elsewhere

STOP and discuss with the user before attempting more fixes. This is not a failed hypothesis — it's a wrong approach.

Red Flags — STOP and Return to Phase 1

If you catch yourself thinking:

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "It's probably X, let me fix that"
• "I don't fully understand but this might work"
• Proposing solutions before tracing data flow
• "One more fix attempt" when already tried 2+

Common Rationalizations

Quick Reference

Attribution

Adapted from obra/superpowers systematic-debugging skill.