Debugging

Debugging

Feedback Loop FIRST: No hypothesis without a repro loop. No fix without root cause. No fix without blast radius scan.

Reference Files

• references/investigation-hygiene.md — investigation discipline, evidence handling
• references/root-cause-playbooks.md — scenario-specific debugging playbooks

Feedback Loop FIRST (Before Any Hypothesis)

A hypothesis without a repro loop is a guess. Before H1, build a fast, deterministic, agent-runnable signal that turns red on the bug.

Construction Ladder (try in rank order, stop at first that works — ordered by loop tightness: earlier rungs are faster and more deterministic)

1. Failing automated test (unit/integration) — best: lives at a seam, reusable as RED
2. curl/HTTP request with asserted response
3. CLI snapshot diff (run command, diff stdout/stderr/exit)
4. Headless browser script (real DOM/runtime crash)
5. Trace replay (recorded request/log/event re-run)
6. Throwaway harness (tiny script calling the suspect function)
7. Property/fuzz check (when failing input is unknown)
8. git bisect run (regression with existing test)
9. Differential old-vs-new (last-good vs HEAD behavior diff)
10. Human-in-the-loop (LAST resort: scripted manual steps)

Tighten the loop — treat it as a product. Once you have a loop, keep tightening:

• Faster? Cache setup, skip unrelated init, narrow the test scope — sub-second beats sub-minute.
• Sharper signal? Assert the exact failing fact, not a noisy superset — never just "didn't crash".
• More deterministic? Pin time, seed RNG, isolate filesystem, freeze network — same input → same red, no drift. A 30-second flaky loop is barely better than none; a 2-second deterministic one is a debugging superpower.

Red-capable completion criteria — the loop is done when you can name one command (a script path, a test invocation, a curl) that you have already run at least once (paste the invocation and its output), and it is:

• Red-capable — drives the actual bug code path and asserts the user's exact symptom (can go red on this bug, green once fixed). Not "runs without erroring".
• Deterministic — same verdict every run (flaky bugs: a pinned, high reproduction rate).
• Fast — seconds, not minutes.
• Agent-runnable — you can run it unattended. No red-capable command, no hypothesis phase. If you catch yourself reading code to build a theory before this command exists, STOP.

Flaky bugs: run in a tight loop (for i in $(seq 1 N); do ...; done), record hit rate (e.g. 3/50), treat raising that rate as loop iteration.

When You Genuinely Cannot Build a Loop

STOP. Do NOT advance to hypothesis. Return BLOCKED with:

• What was tried: each rung attempted and why it failed
• Concrete ask: the one thing that would unblock (env/credential access, captured artifact, permission for temporary instrumentation)

LSP-Powered Root Cause Tracing

Use LSP to trace root causes through the codebase:

• Go to Definition — follow the call chain to where the value is actually set
• Find References — find all callers of a suspect function (blast radius)
• Go to Type Definition — check if the type allows the failing value
• Hover — check types and signatures at the failure site

Don't guess where a value comes from — trace it with LSP. Don't grep for a function name — use Find References to get every caller with type info.

The Four Phases

Phase 1: Root Cause Investigation

1. Understand — expected vs actual, when did it start?
2. Git History — git log --oneline -20 -- <files>, git blame, git diff BASE..HEAD
3. Compounded knowledge — if docs/solutions/debugging/ exists, check for a prior write-up matching this symptom before starting fresh investigation
4. LOG FIRST — collect error logs, stack traces, run failing commands. The error text is the highest-density evidence you will ever get; acting first destroys or masks it.
5. Feedback Loop — build repro signal (construction ladder above). No loop → fail closed.
6. Variant Scan — identify which variant dimensions must keep working (locale, config, env, platform, data shape, concurrency). A fix verified on one variant routinely breaks a sibling variant.

Repro Minimisation: After reproducing the bug, shrink to the smallest scenario that still goes red before forming hypotheses. Cut inputs, callers, config, and environment one at a time. Re-run after each cut. Every remaining element is load-bearing — removing it should make the bug disappear.

Why: A minimal repro shrinks the hypothesis space. The fewer moving parts, the fewer places the bug could hide.

Phase 2: Pattern Analysis

1. Read the code around the failure — not just the failing line, the surrounding logic
2. Check for recent changes — git diff the files involved
3. Look for similar patterns — grep for the same anti-pattern elsewhere
4. Identify the mechanism — not "what's wrong" but "how does the wrong thing happen"

Phase 3: Hypothesis and Testing

Generate 3-5 ranked hypotheses (H1, H2, ...) with 0-100 confidence BEFORE testing any of them — fewer than 3 means you anchored. Rank by explanatory power — which hypothesis explains the most symptoms with the fewest assumptions. Testing the first plausible hypothesis anchors you; generating multiple first prevents anchoring bias and surfaces connections between hypotheses. Proceed to fix only when one reaches 80+.

Hypothesis Quality Criteria:

• States a specific mechanism ("X returns null because Y is not set when Z")
• Predicts a specific test outcome ("if I set Y, X returns the correct value")
• Is falsifiable ("if Y is already set, this hypothesis is wrong")
• Explains ALL observed symptoms, not just the primary one

Instrumentation preference: Each probe must map to a specific prediction. Change one variable at a time. Tool preference:

1. Debugger / REPL inspection if the env supports it — one breakpoint beats ten logs.
2. Targeted logs at the boundaries that distinguish hypotheses.
3. Never "log everything and grep". Tag every debug log with a unique prefix (e.g. [DEBUG-a4f2]) so cleanup is a single grep.

Performance branch: For performance regressions, logs are usually wrong. Establish a baseline measurement first (timing harness, performance.now, profiler, query plan), then bisect. Measure first, fix second.

Hypothesis Confidence Scoring:

When to Restart Investigation: If 3 hypotheses fail, you're pattern-matching, not investigating. Re-read the loop output. Re-trace with LSP. Consider you're looking at the wrong layer.

Phase 4: Implementation

1. RED — failing regression test reproducing the bug (must fail before fix)
2. GREEN — minimal fix (smallest diff, no hardcoding)
3. Blast Radius Scan — search same file for identical anti-patterns, adjacent files for same signature
4. Verify — regression test passes + relevant suite passes
5. Prevention — recommend lint rule, test, type guard, or monitoring. Recommend after the fix is in, not before — you have more information now than when you started.
6. Cleanup — delete throwaway harnesses/prototypes, or move them to a clearly-marked debug location. Grep-remove all [DEBUG-...] tagged instrumentation (must return nothing). Confirm the repro loop no longer fires.

Scenario Playbooks

Read references/root-cause-playbooks.md for scenario-specific guidance:

• State machine bugs (dead states, missing transitions)
• Race conditions (timing-dependent failures)
• Data corruption (cascading from wrong input)
• Performance degradation (regression after change)
• Integration failures (contract mismatch between services)

Debug Attempt Tracking

Track failed hypotheses: [DEBUG-N]: {what was tried} → {result}. After 3 failed hypotheses, set NEEDS_EXTERNAL_RESEARCH: true. After research files provided and still stuck, return BLOCKED.

Causal Chain Gate

Do not propose a fix until you can explain the full causal chain from trigger to symptom with no gaps. "Somehow X leads to Y" is a gap, not an explanation.

Predictions for uncertain links: When the causal chain has an uncertain link, form a prediction — something in a different code path that must also be true if your hypothesis is correct. If the prediction is wrong but the fix "works," you found a symptom fix, not the root cause.

Rationalization Table

Red Flags — STOP and Reconsider

• You're about to make a change without a hypothesis
• You're about to add a try/catch to suppress an error
• You're about to hardcode a value to make a test pass
• You've tried 3 fixes and none worked — you're pattern-matching, not debugging
• You're considering skipping the feedback loop because "the bug is obvious"
• You're about to mark FIXED without a regression test that was RED first
• You're considering weakening an assertion to make the test pass
• You're about to revert a fix and "try something else" without understanding why the fix failed

Pressure Testing

The gates in this skill must hold under pressure — deadline, complexity, "obvious bug" overconfidence. Before trusting a debug cycle:

• Would this gate hold if the user said "just fix it now"?
• Would this gate hold if the bug seemed obvious? The feedback loop gate exists precisely because "obvious" bugs are often wrong diagnoses.
• Would this gate hold at 3am with no sleep? Rationalization tables exist because tired engineers skip process.

Pressure is exactly when these gates pay for themselves — a gate that can be talked out of under pressure was never protecting you.