Systematic Debugging

Systematic Debugging

CRITICAL: This is a diagnostic skill. Investigate first, fix second. Don't guess — gather evidence.

Diagnostic Mindset

• Don't report when healthy. If everything looks fine, say so directly — don't manufacture issues.
• Don't kill or signal processes unless explicitly asked. This is diagnostic only.
• Read errors before guessing causes. The error message is the first evidence.
• Reproduce before fixing. A fix you can't verify is a guess.

Symptom-to-Hypothesis Map

Start here. Match the symptom to candidate hypotheses, then investigate:

Investigation Steps

1. Gather Context

• Read the error message / stack trace carefully. Most errors tell you exactly what's wrong.
• Check recent changes: git log --oneline -10 and git diff — was something just changed?
• Read relevant log files. Use bounded reading: tail the last 50-100 lines, don't read the entire log.

2. Enable Logging If Needed

If the issue isn't visible in current logs:

• Enable debug/verbose logging for the affected component
• Ask the user to reproduce the issue
• Re-read the log after reproduction

3. Form and Test Hypotheses

• Based on symptoms and gathered evidence, form 1-3 hypotheses ranked by likelihood.
• For each hypothesis, identify one specific check that would confirm or eliminate it.
• Run the most discriminating check first.
• If the first hypothesis is disproven, move to the next — don't cling to your initial guess.

4. Narrow Down

• Use Grep with -C 5 context to search for error patterns in logs and code without reading entire files.
• Use head_limit on Grep to avoid drowning in results.
• Only Read full files after Grep confirms they're relevant. Glob (near-zero tokens) → Grep (lightweight) → Read (heavy).
• When debugging spans >5 files, spawn a subagent via the Agent tool for isolated investigation.

5. Fix and Verify

• Make the minimal fix that addresses the root cause, not the symptom.
• Verify the fix by reproducing the original failure scenario — it should now succeed.
• Check for regressions in related functionality.
• If you can't verify (no repro, no test), say so explicitly.

Bounded Log Reading

Logs can be enormous. Never read an entire log file.

• Tail first: Read the last 50-100 lines to find recent errors.
• Grep for severity: Search for [ERROR], [WARN], FATAL, panic, Exception, traceback across the full file.
• Use offset/limit: When you find a relevant line number, read just that region with surrounding context.
• Check file size first: If a log is >1MB, definitely use targeted reads, not full file reads.

Bug Report Format

When reporting findings, use this two-part structure:

Short Summary (1-2 sentences)

What's wrong, which component, and the severity. This is what the reader sees first.

Example: "Auth service returning 500 on /login — null pointer at session.validate()
when session cookie is expired but token is still cached."

Detailed Thread

• Symptom: What was observed (exact error messages, status codes, behavior)
• Root cause: What's actually wrong and why (with evidence — file paths, line numbers, log excerpts)
• Evidence: Commands run and their output (copy-paste, not paraphrased)
• Fix: What was changed or what should be changed (specific files and lines)
• Verification: How the fix was verified, or why it couldn't be verified

Process Diagnostics

When investigating stuck or slow processes:

# List processes with CPU, memory, state, and uptime
ps -axo pid=,pcpu=,rss=,etime=,state=,comm=,command= | grep -E '<process_name>'

# Check child processes
pgrep -lP <pid>

# Sample CPU usage (confirm sustained, not transient)
# Check twice with 1-2s gap

• Filter to relevant processes. Don't dump the entire process table.
• Sample CPU twice to confirm sustained high usage (transient spikes are normal).
• For hung children, get the full command line: ps -p <child_pid> -o command=

CRITICAL REMINDER: Investigate before fixing. Bound your log reads. Report only real issues — don't manufacture problems from healthy systems. Reproduce, then fix, then verify.

Related Skills

• Use codebase-exploration patterns to trace code paths during investigation.
• Use effective-tool-use patterns when running diagnostic commands safely.
• After fixing, apply code-verification patterns to prove the fix works.

Project Customization

If user-config.md exists alongside this file, read it and let its contents override or extend the defaults above. Common customizations:

• Project-specific log locations and formats
• Common failure modes for this system
• Diagnostic commands specific to the tech stack
• Escalation procedures for production issues