Dev-Debugging — Systematic Root Cause Analysis
This skill is the thinking process for fixing bugs. It activates by change surface for errors and diagnoses, and enforces a structured
5-phase methodology for every technical issue — test failures, runtime errors,
build failures, performance regressions, integration bugs.
Boundary: This skill covers how to reason about bugs. For test harness,
reproduction frameworks, and verification tooling, see jaw-dev-testing. For
domain-specific context (API errors, hydration issues, query performance),
consult jaw-dev-backend or jaw-dev-frontend. Comprehension without a defect—understanding an unknown app, API, tool, or codebase—routes to references/logic-analysis.md.
C0/C1 work (small local patches): See jaw-dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.
jaw-dev is canonical: jaw-dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.
jaw-dev-debugging = root cause methodology (the thinking)
jaw-dev-testing = test harness for reproducing/verifying (the tooling)
jaw-dev §2 = summary pointer to this skill (the overview)
Core Principle
Check if the problem is structural before debugging code.
Before a permanent fix, investigate and explain the cause. During an active incident, an already-authorized reversible mitigation may precede full root-cause analysis; follow jaw-dev-devops incident policy and preserve evidence. Diagnosis alone never authorizes a code fix, rollback, production access, or external issue submission.
When to Activate
- Test failures, runtime errors, build failures, performance regressions
- Integration issues (API, database, third-party), CI pipeline failures
- Especially: when under time pressure or when "just one quick fix" seems obvious — that's when methodology matters most
The Phases
Phase 0: Is This a Bug or a Design Problem?
Before debugging code, ask: "Could this be a structural/design issue rather
than a code bug?" Patching symptoms of architectural debt creates an endless
stream of "bugs" that are really design consequences.
Decision Tree — escalate to architecture review if any apply:
| Signal |
Interpretation |
| Same class of bug recurring (3rd time fixing similar issue) |
Design problem — add a constraint at the architecture level |
| Bug spans multiple modules / crosses 2+ boundaries |
Boundary/coupling issue — see jaw-dev-architecture |
| Fix would require changing 3+ files simultaneously |
Likely structural — single-responsibility violation |
| Symptom appears far from cause (error in UI, root in DB layer) |
Tracing/observability gap — instrument boundaries first |
If structural: escalate to architecture review. Do not patch the symptom —
the patch creates the next bug.
Symptom vs Root Cause Fix:
| Symptom |
Likely Patch (wrong) |
Root Cause Fix (right) |
| Request timeout |
Increase timeout to 30s |
Add circuit breaker + fallback |
| OOM crash |
Increase container memory |
Find and fix the memory leak |
| N+1 query performance |
Add a cache layer in front |
Fix the query (eager load / join) |
| Duplicate records |
Add unique constraint + rescue |
Fix the race condition that creates duplicates |
| Flaky test |
Add retry/skip annotation |
Fix shared mutable state between tests |
If none of the above apply — proceed to Phase 1 (it's a code bug, not a
design problem).
Phase 1: Root Cause Investigation
Feedback loop gate: For UI, browser, TUI, visual, streaming, or agent-output bugs,
first create a red-capable loop that can fail before the fix: screenshot/assertion,
recorded terminal bytes, Playwright visual check, log fixture, or a manual repro script
with explicit pass/fail evidence. Do not patch from screenshots alone when a repeatable
probe can be built in reasonable time.
Complete these before attempting any fix:
Read the full error — stack trace, line numbers, error code, surrounding
context. Do not skim. The answer is often in the error message itself.
Reproduce consistently — exact steps to trigger the bug. If intermittent,
document frequency, conditions, and environment state. A bug you cannot
reproduce is a bug you cannot verify as fixed.
Check recent changes — run git log --oneline -10 and git diff. Check
new dependencies, config changes, environment variables. Bugs correlate with
recent changes most of the time.
Trace data flow — where does the bad value originate? Trace backward from
the failure point through the call stack until you find the source. Follow the
full causal chain from trigger → boundary → bad state → failure. Removing the
visible symptom is not a fix unless the defect that creates the bad state is gone.
Instrument component boundaries — for multi-layer systems (API → service →
database, CI → build → deploy), log input/output at each boundary BEFORE
proposing fixes.
Trace-first for distributed/async/agent failures (DEFAULT) — capture the
evidence trail before hypothesizing: request IDs, OpenTelemetry spans/logs,
Playwright traces/videos, and exact agent tool transcripts. For order-dependent,
native, concurrency, or intermittent failures that logs cannot explain, use
time-travel/replay debugging (Microsoft TTD on Windows, rr on Linux) — see
references/tool-guides.md.
For EACH component boundary:
- Log what data enters the component
- Log what data exits the component
- Verify environment/config propagation
Run once → analyze evidence → identify failing layer → investigate THAT layer
Work through these steps; skip only if clearly irrelevant to the problem at hand.
Phase 2: Pattern Analysis
Find working examples — similar working code in the same codebase. If it
worked before, use git bisect to find the breaking commit (see
references/tool-guides.md).
Compare systematically — list every difference between working and broken
code. No matter how small. Resist assuming "that can't matter."
Read reference docs completely — official documentation for the library,
API, or framework involved. Don't skim — read the full relevant section.
Check known issues — GitHub Issues, changelogs, migration guides. Someone
may have hit the same bug. Search with the exact error message.
When the bug depends on third-party library/API/framework behavior, current
error workarounds, upstream issues, changelogs, or migration guides, read the
active jaw-search skill and follow its source-fetch and evidence-status rules
before treating external material as proof.
Phase 3: Hypothesis and Testing
STRICT (DEBUG-RCA-EVIDENCE-01): before investigating any single root-cause
hypothesis, or making any root-cause claim, write at least three orthogonal
hypotheses (H1/H2/H3) and one falsifier for each. Collapse duplicates,
test against disconfirming evidence, and do not claim a root cause until the
competitors have been ruled out by evidence. If fewer than three are plausible,
state why.
Two words in that rule carry the weight. Orthogonal — three restatements of one
theory are one hypothesis, and listing them produces the feeling of rigor with none
of it. Falsifier — a hypothesis you cannot describe a disconfirming observation
for is not testable, so it can never be ruled out and will still be sitting there,
unexcluded, when you claim the winner.
List competing hypotheses first — write at least three plausible root-cause
hypotheses before investigating any single one. Include the evidence that would
support or reject each hypothesis. If fewer than three are plausible, state why.
State the leading hypothesis explicitly — "X is the root cause because
evidence Y shows Z." If you can't articulate it clearly, you don't understand
it yet.
Design a test to disprove — falsification is stronger than confirmation.
What would you expect to see if your hypothesis is wrong?
Test one variable — smallest possible change, one variable at a time.
Never fix multiple things at once.
If it fails → move to another listed hypothesis. Revert the failed change and
start from clean state. Stacking fixes obscures the root cause.
Keep the rejection record — preserve rejected hypotheses and the evidence
that rejected them. The final report must include them, not just the winning cause.
Admit ignorance — "I don't understand X" is a valid finding. Research
further rather than guessing. Record the open question explicitly.
Phase 4: Implementation
STRICT (DEBUG-TOGGLE-PROOF-01): enter implementation only after all three hold:
the captured value matches the hypothesis prediction, the repro repeats, and
toggling the suspected cause off and on removes then restores the bug. Write one
paragraph explaining the causal mechanism before patching.
The toggle is the load-bearing clause, and it is the one usually skipped. A value
that matches the prediction proves correlation; the bug disappearing when you disable
the cause and returning when you re-enable it is what distinguishes the cause from
something that merely co-occurs with it. If the suspected cause cannot be toggled,
say so and treat the diagnosis as unconfirmed rather than quietly upgrading
correlation to causation.
Write a failing test first — the test reproduces the bug. It should fail
before the fix. Use jaw-dev-testing for TDD patterns and test harness setup.
Make the minimal fix — address the root cause, not symptoms. One logical
change only.
Verify: reproduce the repaired behavior and run affected checks at the jaw-dev §3 / jaw-dev-testing risk floor. Respect explicit execution constraints.
Check for similar patterns — does the same bug class exist elsewhere in
the codebase? Search for it. Fix instances within the authorized scope; report additional affected areas instead of silently expanding the patch.
Document — final report and commit message explain root cause AND fix,
including rejected hypotheses and rejection evidence. Not "fixed bug"
but "fix: race condition in session middleware caused by missing await on
Redis write."
Logic Analysis (comprehension without a defect)
For an unfamiliar app, API, tool, or codebase with no defect to repair, use references/logic-analysis.md. Define a falsifiable question, isolate side effects, inventory interfaces, hypothesize, observe, model, and prove one behavior with a minimal authorized client. Report unavailable tools and partial evidence honestly. For example, cli-jaw registers GET /api/events in server.ts; trace its handler in src/routes/events.ts to understand its data-only SSE wire format.
Red Flags — Return to Phase 1
If you catch yourself doing any of these, pause — root cause investigation
was likely skipped.
| Red Flag |
Why It Fails |
| "Quick fix for now, investigate later" |
First fix sets the pattern. Tech debt compounds. You won't investigate later. |
| "Just try changing X and see" |
Guessing guarantees rework. You'll be back here within the hour. |
| "Add multiple changes, run tests" |
Can't isolate cause if multiple variables changed. Revert, change ONE thing. |
| "It's probably X, let me fix that" |
"Probably" without evidence = Phase 1 not done. Go back and trace it. |
| "I don't fully understand but this might work" |
Seeing symptoms ≠ understanding root cause. Your "fix" hides the real bug. |
| "One more fix attempt" (after repeated failures) |
After repeated failures, pause and reassess architecture/assumptions. See escalation below. |
| "It works on my machine" |
Reproduce in the SAME environment as the failure. Local success proves nothing. |
| "Let me add a try/catch around it" |
Suppressing errors is not fixing them. Find WHY it throws. |
Repeated Failure Rule: After repeated failed fix attempts, pause entirely.
Each fix revealing a new problem in a different place is a sign of
architectural issues, not simple bugs. Discuss with the user before
attempting more fixes.
Slop Debugging Patterns
Slop debugging is spray-and-pray: guess, patch, pray, repeat.
| Instead of… |
Use… |
| Proposing fixes before investigation |
Complete Phase 1 checklist first |
| "Might be X" without evidence |
"Evidence shows X because [log/trace/diff]" |
| Multiple simultaneous changes |
One change at a time, revert between attempts |
| Skimming stack traces |
Read every line of stack trace, note line numbers |
Silent catch blocks that suppress errors |
Log with context ([module] error.message), re-throw or handle |
| Modifying failing tests to pass |
Fix the code, not the test — a failing test is evidence |
| Claiming "fixed" without running verification |
Run affected checks and reproduce the repaired behavior at the jaw-dev §3 risk floor |
| Copy-pasting a fix without understanding |
Understand why the fix works, then adapt to your codebase |
| Suppressive try/catch (catch-and-ignore, catch-and-return-null) |
Fix at the source. Boundary catch with logging/re-throw is fine — see jaw-dev-architecture §4. |
| Guessing at types, nulls, or undefined values |
Add diagnostic logging, inspect actual runtime values |
| "It works now" after changing something unrelated |
Correlation ≠ causation — revert the change and test again |
| Letting an AI auto-repair loop (test healer, auto-fix) mask the defect |
Agentic repair aids run only AFTER root cause is understood; keep the failing repro as evidence |
Concrete Debugging Scenarios
Scenario A: API Returns 500
Root cause pattern: Missing input validation lets undefined values propagate into business logic. Instrument controller/service/repository boundaries to find where the bad value enters. Compare with a working endpoint that validates input with a schema. Fix: add schema validation at the entry point, write a test that sends invalid input and expects 400.
Worked example:
curl -i -X POST http://localhost:3000/api/orders \
-H 'content-type: application/json' \
-d '{"sku":"book-1"}'
Observed failure:
HTTP/1.1 500 Internal Server Error
TypeError: Cannot read properties of undefined (reading 'toFixed')
at calculateTotal (src/orders/service.ts:42:21)
at createOrder (src/orders/controller.ts:27:18)
Competing hypotheses before narrowing:
- Request validation allows missing
quantity.
- Controller mapping drops
quantity before service call.
- Repository returns an order row with
quantity = null.
Boundary instrumentation:
DEBUG=orders:* npm run dev
curl -s -X POST http://localhost:3000/api/orders \
-H 'content-type: application/json' \
-d '{"sku":"book-1"}' | jq .
Sample log output:
orders:controller input {"sku":"book-1"}
orders:controller mapped {"sku":"book-1"}
orders:service input {"sku":"book-1"}
orders:repository skipped insert due service error
Rejections: repository-null is rejected because the repository is never reached.
Controller-drop is rejected because controller input already lacks quantity.
Root cause: entry validation accepts a payload missing a required domain field.
Fix at the entry boundary: schema rejects missing quantity; regression test posts
the same payload and expects HTTP 400 with a stable error.code.
Scenario B: React Hydration Mismatch
Root cause pattern: Server renders a value (e.g., date, locale string) that differs from client-side rendering due to environment differences (UTC vs. local timezone). Compare with components that defer environment-dependent rendering to useEffect. Fix: move environment-dependent formatting into a client component.
Scenario C: N+1 Query Performance
Root cause pattern: List endpoint lazy-loads related records per item (1 query + N queries). Enable query logging to count queries, then compare with an endpoint that uses eager loading. Fix: add include/joinedload, write a test asserting bounded query count.
Scenario D: Flaky Test (Intermittent Failure)
Root cause pattern: Test passes in isolation but fails in suite due to shared mutable state (database rows, global variables, uncleared mocks). Compare with stable tests that use transaction rollback in beforeEach/afterEach. Fix: add proper test isolation, then search for other tests missing cleanup.
When to Escalate vs When to Keep Digging
Keep Digging When:
- You have untested hypotheses from Phase 2
- You haven't read the full error message or stack trace
- You haven't checked recent changes (
git log, git diff)
- You haven't found working comparison code yet
- The bug is in YOUR code (not a third-party library)
- You still have untested approaches to try
Escalate When:
- Repeated fix attempts failed — likely architectural; needs human judgment
- Undocumented library behavior — document evidence and propose an upstream report or workaround; posting requires separate authorization
- Environment-specific — requires access you don't have (prod DB, cloud IAM)
- Security-sensitive — follow
jaw-dev-security controls and jaw-dev-devops incident policy; escalate according to the actual risk
- Multi-team dependency — bug is in another team's service or API contract
- Stalled: if investigation stalls, reassess approach
How to Escalate Well
Don't just say "I'm stuck." Provide: symptom (exact error), reproduction
steps, evidence gathered (logs, traces, bisect results), hypotheses
tested (including rejected hypotheses and rejection evidence), remaining hypotheses (untested),
and a recommendation for next steps.
Post-Mortem Discipline
After resolving any bug that:
- Was user/customer-impacting
- Took >1 hour to diagnose
- Involved 3+ failed fix attempts (per postmortem-template.md)
- Revealed a systemic issue (same bug class exists elsewhere)
Fill out references/postmortem-template.md and include it in the PR or commit.
The goal is learning, not blame. Every postmortem must produce at least one
action item that prevents the same class of bug from recurring.
Modular References
| File |
When to Read |
What It Covers |
references/methodologies.md |
Choosing a debug approach |
Five Whys, bisection, differential diagnosis, subtraction, systematic logging |
references/logic-analysis.md |
Understanding a system without a defect |
Controlled observation, incremental model, proof, and honest limits |
references/async-debugging.md |
Concurrency issues |
Race conditions, deadlocks, event loop blocking, promise/callback |
references/tool-guides.md |
Quick cheatsheet |
Node inspector basics, pdb basics, Chrome DevTools, git bisect, DB EXPLAIN |
references/postmortem-template.md |
After resolving a significant incident |
Blameless postmortem template |
references/performance-debugging.md |
Latency, throughput, or memory regressions |
Profiling ladder, measurement discipline |
references/runtimes/node.md |
Node.js / tsx / Bun / Deno |
Phase 0 detection, tsx source-map trap, launch recipes, exec() patterns, silent-failure table, cleanup |
references/runtimes/js/nextjs-react.md |
Next.js 16 / React 19 |
Server-vs-client attach split, hydration mismatch workflow, React Compiler debug, RSC silent failures |
references/runtimes/js/vite-vitest.md |
Vite 8 / Vitest 4 |
Console forwarding to the agent, plugin-transform debug, visual regression, HMR and build-time silent failures |
references/runtimes/js/node-backend.md |
Express 5 / Fastify 5 / NestJS 11 |
Router debug namespaces, lifecycle hooks, schema serialization drops, DI errors, AsyncLocalStorage loss |
references/runtimes/python.md |
Python (CPython 3.9+) |
Attach methods, pdb/ipdb/pudb, pytest, asyncio gotchas, PEP 768 safe attach, py-spy/memray, silent failures |
references/runtimes/rust.md |
Rust |
Hierarchy (dbg! → RUST_LOG → backtrace → gdb/lldb → tokio-console → cargo-expand), Miri UB, silent failures |
references/runtimes/go.md |
Go |
Delve launch/attach, goroutine patterns, race detector, pprof, GODEBUG, silent failures |
references/runtimes/c-cpp.md |
C/C++ |
Sanitizers (ASan/TSan/MSan/UBSan), GDB/LLDB, Valgrind, CMake debug builds, UB silent failures |
references/runtimes/jvm.md |
Java/Kotlin (JVM) |
jcmd live diagnostics, JFR profiling, JDWP/JDB, Kotlin coroutines, GraalVM native-image |
references/runtimes/swift.md |
Swift / iOS |
LLDB, Instruments, Swift concurrency, simulator CLI, crash symbolication |
references/runtimes/ruby.md |
Ruby (3.2+) |
debug gem/rdbg, binding.irb, pry, Rails tools, nil-propagation silent failures |
references/runtimes/beam.md |
Elixir/Erlang (BEAM) |
IEx.pry, Observer, :dbg/recon, supervision hiding, mailbox/atom/binary leaks |
references/tools/playwright.md |
Browser or web-surface bugs |
codegen repro, PWDEBUG, trace viewer, console/network listeners, viewport gotchas |
Read the runtime file for the stack you are actually debugging, not all of them. Each
carries a silent-failure table for its runtime — the failures that produce no
error at all — which is the part hardest to rediscover under time pressure and the
main reason these are worth loading.
Integration with Other Skills
| Skill |
Relationship |
jaw-dev §2 |
Summary of this methodology. This skill is the full version. |
jaw-dev-testing |
Phase 4 "write failing test first" → use jaw-dev-testing for test patterns and harness. jaw-dev-testing provides the tooling; this skill provides the thinking. |
jaw-dev-backend |
Server-side debugging context: API errors, database issues, middleware chains. |
jaw-dev-frontend |
Client-side debugging context: hydration, rendering, DevTools, layout shifts. |
jaw-dev-code-reviewer |
Code review catches bugs before they ship — prevention beats debugging. |
Security-Sensitive Bugs
For security-sensitive bugs (auth bypass, data leak, injection), follow jaw-dev-security controls and jaw-dev-devops incident policy before applying a fix.
Compact Summary
When context is limited, preserve: (1) Phase 0 — is it a bug or a design problem?,
(2) Core principle — investigate before permanent repair; authorized reversible incident mitigation can come first,
(3) phases 0-4 — architecture check → investigate → analyze → hypothesize → implement,
(4) Repeated Failure Rule — after repeated failures, reassess, (5) one variable at a time,
(6) evidence over intuition, (7) failing test first, (8) comprehension without a defect uses references/logic-analysis.md.
1---2name: jaw-dev-debugging3description: "MUST USE for runtime debugging and logic analysis of unknown systems. Covers crashes, wrong output, build/test failures, flaky tests, regressions, integration bugs, and comprehension without a defect. Triggers: debug this, why is X failing, root cause, how does X work, figure out how, reverse engineer, 왜 안 돼, 디버깅, 원인 분석, 로직 파악, 뜯어봐."4metadata:5 source: https://skillmd.com/skills/lidge-jun/jaw-dev-debugging6 short-description: "5-phase systematic root-cause debugging method for real failures in any language."7 keywords: "debug, error, stack trace, root cause, flaky, regression, crash, bisect, trace-first, replay, logic analysis, comprehension, unknown system, reverse engineering"8 last-verified: "2026-07-02"9---1011# Dev-Debugging — Systematic Root Cause Analysis1213This skill is the **thinking process** for fixing bugs. It activates by change surface for errors and diagnoses, and enforces a structured145-phase methodology for every technical issue — test failures, runtime errors,15build failures, performance regressions, integration bugs.1617**Boundary**: This skill covers how to reason about bugs. For test harness,18reproduction frameworks, and verification tooling, see `jaw-dev-testing`. For19domain-specific context (API errors, hydration issues, query performance),20consult `jaw-dev-backend` or `jaw-dev-frontend`. Comprehension without a defect—understanding an unknown app, API, tool, or codebase—routes to `references/logic-analysis.md`.2122> **C0/C1 work (small local patches):** See `jaw-dev` §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.2324> **`jaw-dev` is canonical:** `jaw-dev` §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.2526```27jaw-dev-debugging = root cause methodology (the thinking)28jaw-dev-testing = test harness for reproducing/verifying (the tooling)29jaw-dev §2 = summary pointer to this skill (the overview)30```3132---3334## Core Principle3536Check if the problem is structural before debugging code.37Before a permanent fix, investigate and explain the cause. During an active incident, an already-authorized reversible mitigation may precede full root-cause analysis; follow `jaw-dev-devops` incident policy and preserve evidence. Diagnosis alone never authorizes a code fix, rollback, production access, or external issue submission.3839---4041## When to Activate4243- Test failures, runtime errors, build failures, performance regressions44- Integration issues (API, database, third-party), CI pipeline failures45- **Especially**: when under time pressure or when "just one quick fix" seems obvious — that's when methodology matters most4647---4849## The Phases5051### Phase 0: Is This a Bug or a Design Problem?5253Before debugging code, ask: "Could this be a structural/design issue rather54than a code bug?" Patching symptoms of architectural debt creates an endless55stream of "bugs" that are really design consequences.5657**Decision Tree — escalate to architecture review if any apply:**5859| Signal | Interpretation |60|--------|---------------|61| Same class of bug recurring (3rd time fixing similar issue) | Design problem — add a constraint at the architecture level |62| Bug spans multiple modules / crosses 2+ boundaries | Boundary/coupling issue — see `jaw-dev-architecture` |63| Fix would require changing 3+ files simultaneously | Likely structural — single-responsibility violation |64| Symptom appears far from cause (error in UI, root in DB layer) | Tracing/observability gap — instrument boundaries first |6566**If structural**: escalate to architecture review. Do not patch the symptom —67the patch creates the next bug.6869**Symptom vs Root Cause Fix:**7071| Symptom | Likely Patch (wrong) | Root Cause Fix (right) |72|---------|---------------------|----------------------|73| Request timeout | Increase timeout to 30s | Add circuit breaker + fallback |74| OOM crash | Increase container memory | Find and fix the memory leak |75| N+1 query performance | Add a cache layer in front | Fix the query (eager load / join) |76| Duplicate records | Add unique constraint + rescue | Fix the race condition that creates duplicates |77| Flaky test | Add retry/skip annotation | Fix shared mutable state between tests |7879**If none of the above apply** — proceed to Phase 1 (it's a code bug, not a80design problem).8182---8384### Phase 1: Root Cause Investigation8586**Feedback loop gate:** For UI, browser, TUI, visual, streaming, or agent-output bugs,87first create a red-capable loop that can fail before the fix: screenshot/assertion,88recorded terminal bytes, Playwright visual check, log fixture, or a manual repro script89with explicit pass/fail evidence. Do not patch from screenshots alone when a repeatable90probe can be built in reasonable time.9192**Complete these before attempting any fix:**93941. **Read the full error** — stack trace, line numbers, error code, surrounding95 context. Do not skim. The answer is often in the error message itself.96972. **Reproduce consistently** — exact steps to trigger the bug. If intermittent,98 document frequency, conditions, and environment state. A bug you cannot99 reproduce is a bug you cannot verify as fixed.1001013. **Check recent changes** — run `git log --oneline -10` and `git diff`. Check102 new dependencies, config changes, environment variables. Bugs correlate with103 recent changes most of the time.1041054. **Trace data flow** — where does the bad value originate? Trace backward from106 the failure point through the call stack until you find the source. Follow the107 full causal chain from trigger → boundary → bad state → failure. Removing the108 visible symptom is not a fix unless the defect that creates the bad state is gone.1091105. **Instrument component boundaries** — for multi-layer systems (API → service →111 database, CI → build → deploy), log input/output at each boundary BEFORE112 proposing fixes.1131146. **Trace-first for distributed/async/agent failures (DEFAULT)** — capture the115 evidence trail before hypothesizing: request IDs, OpenTelemetry spans/logs,116 Playwright traces/videos, and exact agent tool transcripts. For order-dependent,117 native, concurrency, or intermittent failures that logs cannot explain, use118 time-travel/replay debugging (Microsoft TTD on Windows, rr on Linux) — see119 `references/tool-guides.md`.120121```122For EACH component boundary:123 - Log what data enters the component124 - Log what data exits the component125 - Verify environment/config propagation126Run once → analyze evidence → identify failing layer → investigate THAT layer127```128129Work through these steps; skip only if clearly irrelevant to the problem at hand.130131### Phase 2: Pattern Analysis1321331. **Find working examples** — similar working code in the same codebase. If it134 worked before, use `git bisect` to find the breaking commit (see135 `references/tool-guides.md`).1361372. **Compare systematically** — list every difference between working and broken138 code. No matter how small. Resist assuming "that can't matter."1391403. **Read reference docs completely** — official documentation for the library,141 API, or framework involved. Don't skim — read the full relevant section.1421434. **Check known issues** — GitHub Issues, changelogs, migration guides. Someone144 may have hit the same bug. Search with the exact error message.145146When the bug depends on third-party library/API/framework behavior, current147error workarounds, upstream issues, changelogs, or migration guides, read the148active `jaw-search` skill and follow its source-fetch and evidence-status rules149before treating external material as proof.150151### Phase 3: Hypothesis and Testing152153**STRICT (DEBUG-RCA-EVIDENCE-01):** before investigating any single root-cause154hypothesis, or making any root-cause claim, write at least three **orthogonal**155hypotheses (`H1`/`H2`/`H3`) and **one falsifier for each**. Collapse duplicates,156test against disconfirming evidence, and do not claim a root cause until the157competitors have been ruled out **by evidence**. If fewer than three are plausible,158state why.159160Two words in that rule carry the weight. **Orthogonal** — three restatements of one161theory are one hypothesis, and listing them produces the feeling of rigor with none162of it. **Falsifier** — a hypothesis you cannot describe a disconfirming observation163for is not testable, so it can never be ruled out and will still be sitting there,164unexcluded, when you claim the winner.1651661. **List competing hypotheses first** — write at least three plausible root-cause167 hypotheses before investigating any single one. Include the evidence that would168 support or reject each hypothesis. If fewer than three are plausible, state why.1691702. **State the leading hypothesis explicitly** — "X is the root cause because171 evidence Y shows Z." If you can't articulate it clearly, you don't understand172 it yet.1731743. **Design a test to disprove** — falsification is stronger than confirmation.175 What would you expect to see if your hypothesis is wrong?1761774. **Test one variable** — smallest possible change, one variable at a time.178 Never fix multiple things at once.1791805. **If it fails** → move to another listed hypothesis. Revert the failed change and181 start from clean state. Stacking fixes obscures the root cause.1821836. **Keep the rejection record** — preserve rejected hypotheses and the evidence184 that rejected them. The final report must include them, not just the winning cause.1851867. **Admit ignorance** — "I don't understand X" is a valid finding. Research187 further rather than guessing. Record the open question explicitly.188189### Phase 4: Implementation190191**STRICT (DEBUG-TOGGLE-PROOF-01):** enter implementation only after all three hold:192the captured value matches the hypothesis prediction, the repro repeats, and193**toggling the suspected cause off and on removes then restores the bug.** Write one194paragraph explaining the causal mechanism before patching.195196The toggle is the load-bearing clause, and it is the one usually skipped. A value197that matches the prediction proves correlation; the bug disappearing when you disable198the cause and returning when you re-enable it is what distinguishes the cause from199something that merely co-occurs with it. If the suspected cause cannot be toggled,200say so and treat the diagnosis as unconfirmed rather than quietly upgrading201correlation to causation.2022031. **Write a failing test first** — the test reproduces the bug. It should fail204 before the fix. Use `jaw-dev-testing` for TDD patterns and test harness setup.2052062. **Make the minimal fix** — address the root cause, not symptoms. One logical207 change only.2082093. **Verify**: reproduce the repaired behavior and run affected checks at the `jaw-dev` §3 / `jaw-dev-testing` risk floor. Respect explicit execution constraints.2102114. **Check for similar patterns** — does the same bug class exist elsewhere in212 the codebase? Search for it. Fix instances within the authorized scope; report additional affected areas instead of silently expanding the patch.2132145. **Document** — final report and commit message explain root cause AND fix,215 including rejected hypotheses and rejection evidence. Not "fixed bug"216 but "fix: race condition in session middleware caused by missing await on217 Redis write."218219---220221## Logic Analysis (comprehension without a defect)222223For an unfamiliar app, API, tool, or codebase with no defect to repair, use `references/logic-analysis.md`. Define a falsifiable question, isolate side effects, inventory interfaces, hypothesize, observe, model, and prove one behavior with a minimal authorized client. Report unavailable tools and partial evidence honestly. For example, cli-jaw registers `GET /api/events` in `server.ts`; trace its handler in `src/routes/events.ts` to understand its data-only SSE wire format.224225---226227## Red Flags — Return to Phase 1228229If you catch yourself doing any of these, pause — root cause investigation230was likely skipped.231232| Red Flag | Why It Fails |233|----------|-------------|234| "Quick fix for now, investigate later" | First fix sets the pattern. Tech debt compounds. You won't investigate later. |235| "Just try changing X and see" | Guessing guarantees rework. You'll be back here within the hour. |236| "Add multiple changes, run tests" | Can't isolate cause if multiple variables changed. Revert, change ONE thing. |237| "It's probably X, let me fix that" | "Probably" without evidence = Phase 1 not done. Go back and trace it. |238| "I don't fully understand but this might work" | Seeing symptoms ≠ understanding root cause. Your "fix" hides the real bug. |239| "One more fix attempt" (after repeated failures) | After repeated failures, pause and reassess architecture/assumptions. See escalation below. |240| "It works on my machine" | Reproduce in the SAME environment as the failure. Local success proves nothing. |241| "Let me add a try/catch around it" | Suppressing errors is not fixing them. Find WHY it throws. |242243**Repeated Failure Rule**: After repeated failed fix attempts, pause entirely.244Each fix revealing a new problem in a different place is a sign of245**architectural issues**, not simple bugs. Discuss with the user before246attempting more fixes.247248---249250## Slop Debugging Patterns251252Slop debugging is spray-and-pray: guess, patch, pray, repeat.253254| Instead of… | Use… |255|-------------|------|256| Proposing fixes before investigation | Complete Phase 1 checklist first |257| "Might be X" without evidence | "Evidence shows X because [log/trace/diff]" |258| Multiple simultaneous changes | One change at a time, revert between attempts |259| Skimming stack traces | Read every line of stack trace, note line numbers |260| Silent `catch` blocks that suppress errors | Log with context (`[module] error.message`), re-throw or handle |261| Modifying failing tests to pass | Fix the code, not the test — a failing test is evidence |262| Claiming "fixed" without running verification | Run affected checks and reproduce the repaired behavior at the `jaw-dev` §3 risk floor |263| Copy-pasting a fix without understanding | Understand why the fix works, then adapt to your codebase |264| Suppressive try/catch (catch-and-ignore, catch-and-return-null) | Fix at the source. Boundary catch with logging/re-throw is fine — see jaw-dev-architecture §4. |265| Guessing at types, nulls, or undefined values | Add diagnostic logging, inspect actual runtime values |266| "It works now" after changing something unrelated | Correlation ≠ causation — revert the change and test again |267| Letting an AI auto-repair loop (test healer, auto-fix) mask the defect | Agentic repair aids run only AFTER root cause is understood; keep the failing repro as evidence |268269---270271## Concrete Debugging Scenarios272273### Scenario A: API Returns 500274275Root cause pattern: Missing input validation lets undefined values propagate into business logic. Instrument controller/service/repository boundaries to find where the bad value enters. Compare with a working endpoint that validates input with a schema. Fix: add schema validation at the entry point, write a test that sends invalid input and expects 400.276277Worked example:278279```bash280curl -i -X POST http://localhost:3000/api/orders \281 -H 'content-type: application/json' \282 -d '{"sku":"book-1"}'283```284285Observed failure:286287```text288HTTP/1.1 500 Internal Server Error289TypeError: Cannot read properties of undefined (reading 'toFixed')290 at calculateTotal (src/orders/service.ts:42:21)291 at createOrder (src/orders/controller.ts:27:18)292```293294Competing hypotheses before narrowing:2952961. Request validation allows missing `quantity`.2972. Controller mapping drops `quantity` before service call.2983. Repository returns an order row with `quantity = null`.299300Boundary instrumentation:301302```bash303DEBUG=orders:* npm run dev304curl -s -X POST http://localhost:3000/api/orders \305 -H 'content-type: application/json' \306 -d '{"sku":"book-1"}' | jq .307```308309Sample log output:310311```text312orders:controller input {"sku":"book-1"}313orders:controller mapped {"sku":"book-1"}314orders:service input {"sku":"book-1"}315orders:repository skipped insert due service error316```317318Rejections: repository-null is rejected because the repository is never reached.319Controller-drop is rejected because controller input already lacks `quantity`.320Root cause: entry validation accepts a payload missing a required domain field.321Fix at the entry boundary: schema rejects missing `quantity`; regression test posts322the same payload and expects HTTP 400 with a stable `error.code`.323324### Scenario B: React Hydration Mismatch325326Root cause pattern: Server renders a value (e.g., date, locale string) that differs from client-side rendering due to environment differences (UTC vs. local timezone). Compare with components that defer environment-dependent rendering to useEffect. Fix: move environment-dependent formatting into a client component.327328### Scenario C: N+1 Query Performance329330Root cause pattern: List endpoint lazy-loads related records per item (1 query + N queries). Enable query logging to count queries, then compare with an endpoint that uses eager loading. Fix: add include/joinedload, write a test asserting bounded query count.331332### Scenario D: Flaky Test (Intermittent Failure)333334Root cause pattern: Test passes in isolation but fails in suite due to shared mutable state (database rows, global variables, uncleared mocks). Compare with stable tests that use transaction rollback in beforeEach/afterEach. Fix: add proper test isolation, then search for other tests missing cleanup.335336---337338## When to Escalate vs When to Keep Digging339340### Keep Digging When:341342- You have untested hypotheses from Phase 2343- You haven't read the full error message or stack trace344- You haven't checked recent changes (`git log`, `git diff`)345- You haven't found working comparison code yet346- The bug is in YOUR code (not a third-party library)347- You still have untested approaches to try348349### Escalate When:350351- **Repeated fix attempts failed** — likely architectural; needs human judgment352- **Undocumented library behavior** — document evidence and propose an upstream report or workaround; posting requires separate authorization353- **Environment-specific** — requires access you don't have (prod DB, cloud IAM)354- **Security-sensitive** — follow `jaw-dev-security` controls and `jaw-dev-devops` incident policy; escalate according to the actual risk355- **Multi-team dependency** — bug is in another team's service or API contract356- **Stalled**: if investigation stalls, reassess approach357358### How to Escalate Well359360Don't just say "I'm stuck." Provide: **symptom** (exact error), **reproduction361steps**, **evidence gathered** (logs, traces, bisect results), **hypotheses362tested** (including rejected hypotheses and rejection evidence), **remaining hypotheses** (untested),363and a **recommendation** for next steps.364365---366367## Post-Mortem Discipline368369After resolving any bug that:370- Was user/customer-impacting371- Took >1 hour to diagnose372- Involved 3+ failed fix attempts (per postmortem-template.md)373- Revealed a systemic issue (same bug class exists elsewhere)374375Fill out `references/postmortem-template.md` and include it in the PR or commit.376The goal is **learning, not blame**. Every postmortem must produce at least one377action item that prevents the same class of bug from recurring.378379---380381## Modular References382383| File | When to Read | What It Covers |384|------|-------------|----------------|385| `references/methodologies.md` | Choosing a debug approach | Five Whys, bisection, differential diagnosis, subtraction, systematic logging |386| `references/logic-analysis.md` | Understanding a system without a defect | Controlled observation, incremental model, proof, and honest limits |387| `references/async-debugging.md` | Concurrency issues | Race conditions, deadlocks, event loop blocking, promise/callback |388| `references/tool-guides.md` | Quick cheatsheet | Node inspector basics, pdb basics, Chrome DevTools, git bisect, DB EXPLAIN |389| `references/postmortem-template.md` | After resolving a significant incident | Blameless postmortem template |390| `references/performance-debugging.md` | Latency, throughput, or memory regressions | Profiling ladder, measurement discipline |391| `references/runtimes/node.md` | Node.js / tsx / Bun / Deno | Phase 0 detection, tsx source-map trap, launch recipes, `exec()` patterns, silent-failure table, cleanup |392| `references/runtimes/js/nextjs-react.md` | Next.js 16 / React 19 | Server-vs-client attach split, hydration mismatch workflow, React Compiler debug, RSC silent failures |393| `references/runtimes/js/vite-vitest.md` | Vite 8 / Vitest 4 | Console forwarding to the agent, plugin-transform debug, visual regression, HMR and build-time silent failures |394| `references/runtimes/js/node-backend.md` | Express 5 / Fastify 5 / NestJS 11 | Router debug namespaces, lifecycle hooks, schema serialization drops, DI errors, AsyncLocalStorage loss |395| `references/runtimes/python.md` | Python (CPython 3.9+) | Attach methods, pdb/ipdb/pudb, pytest, asyncio gotchas, PEP 768 safe attach, py-spy/memray, silent failures |396| `references/runtimes/rust.md` | Rust | Hierarchy (`dbg!` → `RUST_LOG` → backtrace → gdb/lldb → tokio-console → cargo-expand), Miri UB, silent failures |397| `references/runtimes/go.md` | Go | Delve launch/attach, goroutine patterns, race detector, pprof, GODEBUG, silent failures |398| `references/runtimes/c-cpp.md` | C/C++ | Sanitizers (ASan/TSan/MSan/UBSan), GDB/LLDB, Valgrind, CMake debug builds, UB silent failures |399| `references/runtimes/jvm.md` | Java/Kotlin (JVM) | jcmd live diagnostics, JFR profiling, JDWP/JDB, Kotlin coroutines, GraalVM native-image |400| `references/runtimes/swift.md` | Swift / iOS | LLDB, Instruments, Swift concurrency, simulator CLI, crash symbolication |401| `references/runtimes/ruby.md` | Ruby (3.2+) | debug gem/rdbg, `binding.irb`, pry, Rails tools, nil-propagation silent failures |402| `references/runtimes/beam.md` | Elixir/Erlang (BEAM) | `IEx.pry`, Observer, `:dbg`/recon, supervision hiding, mailbox/atom/binary leaks |403| `references/tools/playwright.md` | Browser or web-surface bugs | codegen repro, PWDEBUG, trace viewer, console/network listeners, viewport gotchas |404405Read the runtime file for the stack you are actually debugging, not all of them. Each406carries a **silent-failure table** for its runtime — the failures that produce no407error at all — which is the part hardest to rediscover under time pressure and the408main reason these are worth loading.409410411---412413## Integration with Other Skills414415| Skill | Relationship |416|-------|-------------|417| `jaw-dev` §2 | Summary of this methodology. This skill is the full version. |418| `jaw-dev-testing` | Phase 4 "write failing test first" → use `jaw-dev-testing` for test patterns and harness. `jaw-dev-testing` provides the tooling; this skill provides the thinking. |419| `jaw-dev-backend` | Server-side debugging context: API errors, database issues, middleware chains. |420| `jaw-dev-frontend` | Client-side debugging context: hydration, rendering, DevTools, layout shifts. |421| `jaw-dev-code-reviewer` | Code review catches bugs before they ship — prevention beats debugging. |422423---424425## Security-Sensitive Bugs426427For security-sensitive bugs (auth bypass, data leak, injection), follow `jaw-dev-security` controls and `jaw-dev-devops` incident policy before applying a fix.428429---430431## Compact Summary432433When context is limited, preserve: (1) Phase 0 — is it a bug or a design problem?,434(2) Core principle — investigate before permanent repair; authorized reversible incident mitigation can come first,435(3) phases 0-4 — architecture check → investigate → analyze → hypothesize → implement,436(4) Repeated Failure Rule — after repeated failures, reassess, (5) one variable at a time,437(6) evidence over intuition, (7) failing test first, (8) comprehension without a defect uses `references/logic-analysis.md`.