Systematic Debug
Structured, hypothesis-driven debugging. No guessing. No "let me just try this." Every action has a reason, every observation narrows the search space.
When To Use
- A test fails and the reason isn't obvious
- A fix introduced a regression
- Behavior doesn't match expectations
- A trailing verification agent found something wrong
- "It works locally but not in CI/prod"
- Something that was working stopped working
Philosophy
The goal is not to fix the bug. The goal is to understand the bug. The fix follows naturally.
Rules:
- Never change code to "see if this helps" — change code because you know WHY it will help
- Read before guessing. Most bugs are obvious once you read the right 20 lines of code
- Shrink the search space with each step. If a step doesn't eliminate possibilities, it was wasted
- Reproduce first. If you can't trigger it, you can't verify your fix
Phase 1: Stabilize (What exactly is broken?)
Before investigating, nail down the observable symptoms.
Collect:
- Error output — exact error message, stack trace, log lines (copy verbatim, don't paraphrase)
- Expected behavior — what SHOULD happen?
- Actual behavior — what DOES happen?
- Reproduction — can you trigger it reliably? What's the minimal reproduction?
- Recency — when did this last work? What changed since then?
If recency is known: git log --oneline from the last known-good state. The diff between then and now is your search space.
If recency is unknown: Skip ahead, but flag this — you'll need to bisect later.
Output: A one-paragraph problem statement. Example:
"The
storychief-push-drafttask returns 403 after the auth middleware refactor in commit abc123. It worked on commit def456. The error isForbidden: invalid API scope. Expected: draft pushes successfully."
Phase 2: Narrow (Where is the bug?)
Systematically reduce the search space. Pick the strategy that fits:
Strategy A: Recent Change (you know what changed)
- Read the diff (
git diff <last-good>..<current>or specific files) - For each changed file, ask: "Could this change cause the observed symptom?"
- Rank changes by likelihood (closest to the error path first)
- Read the top 2-3 suspect files in full (not just the diff — context matters)
Strategy B: Stack Trace (you have one)
- Start at the TOP of the stack trace (the throw/error site)
- Read that file at that line. Understand what triggered the error
- Walk UP the call chain — what passed bad data? What state was wrong?
- The bug is usually 1-3 frames up from where the error surfaces
Strategy C: Bisect (no stack trace, no obvious change)
- Find a known-good state (commit, config, input)
- Find the known-bad state (current)
- Test the midpoint
- Repeat until you've isolated the breaking change
- For git:
git log --oneline <good>..<bad>then manually test the midpoint commit
Strategy D: Input/Output Trace (behavior is wrong, no error)
- Identify the entry point (API call, function call, event trigger)
- Trace the data flow through each transformation step
- At each step: "Is the input correct? Is the output correct?"
- The bug is at the first step where output diverges from expected
Output: A specific location (file:line) or narrow area (2-3 files) where the bug lives.
Phase 3: Hypothesize & Test (Why is it broken?)
Now you have a location. Form a hypothesis and test it — without changing production code yet.
Hypothesis format:
"The bug is caused by [specific mechanism] in [file:line]. This happens because [chain of causation]. If I'm right, then [testable prediction]."
Testing a hypothesis (pick one):
- Read test: Re-read the code path with your hypothesis in mind. Does every step confirm it?
- Log test: Add a temporary log/print at the suspect location. Run the reproduction. Does the output confirm your hypothesis?
- Minimal reproduction: Write a small test that isolates just the suspect behavior. Does it fail the way you predict?
- Counter-test: If your hypothesis is right, what ELSE should be broken? Check those cases.
If the hypothesis is CONFIRMED: Move to Phase 4.
If the hypothesis is REJECTED:
- What did you learn? What possibilities were eliminated?
- Form the next hypothesis based on the narrowed search space
- Do NOT loop more than 3 hypotheses without stepping back to Phase 2 to re-narrow
Phase 4: Fix & Verify (Make it right, prove it's right)
Fix:
- Make the minimal change that addresses the root cause (not the symptom)
- If the fix is more than ~20 lines, pause — you may be fixing the wrong thing or over-engineering
- Do NOT fix adjacent issues you noticed. One fix per debug cycle. File other issues separately.
Verify:
- Run the original reproduction — does it pass now?
- Run the full test suite — did the fix break anything else?
- If the bug was a regression, check the specific area that originally triggered it
- Explain the fix in one sentence. If you can't, you may not fully understand it yet.
Fix statement format:
"The bug was [root cause] in [file:line]. It was caused by [mechanism]. Fixed by [what you changed]. Verified by [what you ran]."
Sub-Agent Mode
When this methodology is used by a sub-agent (e.g., a trailing verification agent in Ralph Review or audit):
The sub-agent MUST:
- Include the full problem statement (Phase 1 output) in its report
- Include the specific location (Phase 2 output)
- Include the confirmed hypothesis (Phase 3 output)
- Include the fix statement (Phase 4 output)
- Never say "I fixed it" without the fix statement
The parent agent MUST:
- Not accept "fixed" without a fix statement
- Verify the fix independently (run tests, read the diff)
- If the sub-agent looped 3+ hypotheses without resolution, escalate to the user
Anti-Patterns (Banned)
| Pattern | Why It's Banned | Do This Instead |
|---|---|---|
| "Let me try changing X and see if it works" | Guess-and-check doesn't build understanding | Form a hypothesis first, test it without code changes |
| Changing multiple things at once | Can't tell which change fixed it (or broke something else) | One change per cycle |
| "It works now, not sure why" | You'll be back here next week | Keep investigating until you can write the fix statement |
| Searching the whole codebase for the error string | Unfocused; wastes time | Use Phase 2 strategies to narrow first |
| "Probably a race condition / caching issue" | These are cop-out hypotheses | Be specific: which race? Which cache? What state? |
| Deleting and rewriting the broken code | Hides the bug instead of understanding it | Understand first, then decide if rewrite is warranted |