Debug Ops

Debug Operations

Systematic debugging methodology with language-specific tooling and common scenario playbooks.

Bug Classification Decision Tree

Bug Report / Symptom
│
├─ Crash
│  ├─ Segfault / Access Violation
│  │  └─ Check: null pointer, buffer overflow, use-after-free, stack overflow
│  ├─ Panic / Fatal Error
│  │  └─ Check: assertion failure, unrecoverable state, out-of-memory
│  └─ Unhandled Exception
│     └─ Check: missing error handler, unexpected input type, network failure
│
├─ Hang
│  ├─ Deadlock
│  │  └─ Check: lock ordering, mutex contention, channel blocking
│  ├─ Infinite Loop
│  │  └─ Check: loop termination condition, counter overflow, recursive call
│  └─ Blocked I/O
│     └─ Check: network timeout, DNS resolution, disk full, file lock
│
├─ Wrong Output
│  ├─ Logic Error
│  │  └─ Check: operator precedence, boundary conditions, boolean logic
│  ├─ Data Corruption
│  │  └─ Check: concurrent mutation, encoding mismatch, truncation
│  └─ Off-by-One
│     └─ Check: loop bounds, array indexing, fence-post errors
│
├─ Performance
│  ├─ Slow Queries
│  │  └─ Check: missing index, N+1 queries, full table scan, lock wait
│  ├─ Memory Bloat
│  │  └─ Check: cache without eviction, leaked references, large allocations
│  └─ CPU Spikes
│     └─ Check: hot loops, regex backtracking, excessive GC, busy-wait
│
└─ Intermittent
   ├─ Race Condition
   │  └─ Check: shared mutable state, read-modify-write, check-then-act
   ├─ Timing-Dependent
   │  └─ Check: timeout values, clock skew, event ordering assumptions
   └─ Environment-Specific
      └─ Check: OS differences, locale, timezone, file system case sensitivity

Systematic Debugging Workflow

Six-step process from symptom to prevention:

Step 1: Reproduce

Confirm the bug exists and create a reliable reproduction. A bug you cannot reproduce is a bug you cannot confidently fix. Capture exact inputs, environment, and sequence of operations.

Step 2: Isolate

Narrow the fault to the smallest possible scope. Use binary search (git bisect, commenting out code halves), stubs, feature flags, and environment isolation to eliminate innocent code.

Step 3: Identify

Find the root cause, not just the proximate trigger. Use the 5 Whys technique, trace execution, inspect state at key points. Distinguish between the symptom and the underlying defect.

Step 4: Fix

Apply the minimal correct change that addresses the root cause. Avoid shotgun debugging (changing multiple things at once). Understand why the fix works, not just that it works.

Step 5: Verify

Confirm the fix resolves the original issue without introducing regressions. Re-run the original reproduction case. Run the full test suite. Test edge cases related to the fix.

Step 6: Prevent

Add a regression test. Update documentation or runbooks if applicable. Consider whether the same class of bug could exist elsewhere. Share findings with the team.

Reproduction Checklist

[ ] Minimal reproduction steps documented (numbered, unambiguous)
[ ] Environment captured (OS, runtime version, dependencies, config)
[ ] Exact inputs recorded (request payload, CLI args, file contents)
[ ] Timing sensitivity assessed (does it fail only under load? after delay?)
[ ] Single-threaded reproduction attempted (eliminates concurrency noise)
[ ] Reproduction automated as script or test case
[ ] Confirmed reproduction is deterministic (fails N/N attempts)
[ ] Identified whether reproduction requires specific data/state

Isolation Techniques Quick Reference

Root Cause Analysis Template

5 Whys Example

Problem: API returns 500 error on user login

1. Why? → The database query throws a timeout exception
2. Why? → The users table scan takes >30 seconds
3. Why? → There is no index on the email column
4. Why? → The migration that adds the index was never run in production
5. Why? → The deployment script skips migrations when the --fast flag is used

Root cause: Deployment script's --fast flag bypasses migrations
Fix: Remove --fast flag behavior that skips migrations, add migration check to health endpoint
Prevention: CI check that verifies all migrations are applied after deployment

Fault Tree Basics

                    [System Failure]
                    /              \
            [Hardware]          [Software]
            /       \           /        \
       [Disk]    [Memory]  [Config]   [Code Bug]
                              |          |
                         [Missing    [Race in
                          env var]    worker pool]

Work from the top (observed failure) down to leaves (root causes). Each branch is an AND/OR gate -- AND means all children must be true, OR means any one child suffices.

Language-Specific Debugger Quick Reference

Quick Debug Snippets

// Node.js: drop into debugger at this point
debugger;

// Node.js: conditional breakpoint
if (user.id === 'problem-user') debugger;

# Python: drop into debugger at this point
breakpoint()

# Python: conditional breakpoint
if user_id == 'problem-user':
    breakpoint()

// Go: print goroutine stacks (send SIGQUIT or SIGABRT)
// kill -QUIT <pid>
// Or in code:
import "runtime/debug"
debug.PrintStack()

// Rust: enable full backtraces
// RUST_BACKTRACE=1 cargo run
// RUST_BACKTRACE=full cargo run

Log-Based Debugging Patterns

Strategic Logging

Place logs at decision points, not just error paths:

[ENTRY] function_name(args_summary)     -- entering the function
[STATE] key_variable=value              -- state at critical decision point
[BRANCH] taking path X because Y       -- which branch and why
[EXIT] function_name -> result_summary  -- leaving the function
[ERROR] operation failed: detail        -- error with context

Correlation IDs

Trace a single request across services:

# Generate at entry point, propagate through all calls
X-Request-ID: 550e8400-e29b-41d4-a716-446655440000

# Search across all service logs
rg "550e8400-e29b-41d4-a716-446655440000" /var/log/services/

Timeline Reconstruction

# Merge and sort logs from multiple sources by timestamp
sort -t' ' -k1,2 service-a.log service-b.log service-c.log > timeline.log

# Find gaps in activity (potential hang/block)
awk '{print $1, $2}' timeline.log | uniq -c | sort -rn | head -20

Structured Log Queries

# jq queries on JSON logs
# Find all errors for a specific user
jq 'select(.level == "error" and .user_id == "u123")' app.log

# Get timing distribution for slow requests
jq 'select(.duration_ms > 1000) | .duration_ms' app.log | sort -n

# Count errors by type
jq -r 'select(.level == "error") | .error_type' app.log | sort | uniq -c | sort -rn

Common Gotchas

Reference Files

See Also

• testing-ops -- Write tests to prevent bugs from recurring
• security-ops -- Security-specific debugging (auth failures, injection, CSRF)
• monitoring-ops -- Production observability, alerting, dashboards
• code-stats -- Measure code complexity and identify bug-prone areas
• container-orchestration -- Docker and Kubernetes debugging context
• git-ops -- Git bisect workflow and history investigation