Dev Loop Debugging Strategy

Debugging Strategy

Purpose

Apply systematic debugging methodologies — root cause analysis, scientific method for debugging, log analysis, and tool-assisted tracing — to identify, isolate, and resolve software defects efficiently.

Agent Protocol

Trigger

Exact user phrases: "debugging", "debug strategy", "how to debug", "root cause", "bug reproduction", "debugging methodology", "log analysis", "troubleshooting", "debug this", "why is this broken".

Input Context

• Environment (local dev, CI, staging, production)
• Bug characteristics (reproducible, intermittent, regression, performance, crash)
• Available tools (debugger, profiler, logs, APM, metrics)
• Language and framework (affects available debugging tools)
• Recent changes (deployments, dependency updates, config changes)
• Error symptoms (error message, stack trace, incorrect output, crash dump)

Output Artifact

Debugging plan with hypothesis, isolation strategy, reproduction steps, and root cause analysis.

Completion Criteria

• Bug reproduction steps documented and confirmed
• Initial hypothesis formed with expected vs actual behavior
• Debugging tooling selected and configured
• Isolation strategy applied (binary search, eliminate variables)
• Root cause identified and documented
• Fix implemented and verified
• Regression test added to prevent recurrence
• Postmortem notes documented (if warranted)

Max Response Length

200 lines.

Framework/Methodology

Debugging Decision Tree

What type of bug?
├── Application crash (exception, segfault, panic)
│   → Stack trace analysis → crash dump → core dump → last known good
├── Incorrect behavior (wrong output, wrong state)
│   → Scientific method: hypothesis → experiment → observe → conclude
│   → Binary search on commits: git bisect
├── Intermittent failure (race condition, timing)
│   → Stress testing → logging → thread/async analysis → happens-before
├── Performance regression (slow response, high CPU/memory)
│   → Profiler → flame graph → heap dump → load testing
├── Integration failure (API, database, network)
│   → Request/response logging → mock → dependency isolation
└── Environmental issue (wrong config, missing dependency)
    → Config diff → environment comparison → fresh install

The Scientific Method for Debugging

1. Observe:   What is the symptom? When does it happen? What changed?
2. Hypothesize: What could cause this? Rank by probability.
3. Predict:   If hypothesis is correct, what else would we expect?
4. Experiment: Design a test to validate or invalidate the hypothesis.
5. Analyze:   Did the result match the prediction?
6. Conclude:  If validated → fix. If invalidated → refine hypothesis (goto 2).

Debugging Levels (Eskil Steenberg's Model)

Level 0: No methodology — random changes hoping something works
Level 1: Observation — reading error messages, stack traces
Level 2: Isolation — binary search, eliminate variables
Level 3: Tooling — debuggers, profilers, log analysis
Level 4: Scientific method — hypothesis-driven, systematic
Level 5: Prevention — tests, type systems, assertions prevent bugs

Workflow

Step 1: Reproduce (Always the First Step)

Create a minimal, deterministic reproduction:

reproduction_steps:
  environment: "Node.js 20, macOS 14.5, PostgreSQL 16"
  prerequisites:
    - "npm install"
    - "docker compose up -d db"
  steps:
    - "Run: npm run dev"
    - "Navigate to /settings"
    - "Click 'Save' with empty name field"
    - "Expected: Validation error shown"
    - "Actual: Page crashes with TypeError"
  frequency: "100% reproducible"
  regression: true
  last_known_good: "v1.2.0"

If not reproducible:

• Check environment differences (local vs CI vs production)
• Check data differences (production data may trigger edge case)
• Check timing (race condition, async timing)
• Add extensive logging
• Use session replay tools (FullStory, LogRocket, Sentry)

Step 2: Gather Data

# Stack trace analysis
# Look for: YOUR code in the trace (not third-party), null reference, type error

# Binary search commits (git bisect)
git bisect start
git bisect bad          # Current commit is broken
git bisect good v1.0.0  # Last known good
# Git checks out the midpoint; test it:
npm test
git bisect good         # or git bisect bad
# Repeat until commit identified

# Log analysis
kubectl logs -l app=myapp --tail=100 --since=10m > logs.txt
# Look for: error, exception, fatal, timeout, 500, stack trace, correlation ID

# Thread dump analysis (Java)
jstack <pid> > threaddump.txt
# Look for: BLOCKED threads, DEADLOCK, stuck threads

# Core dump analysis (C/C++/Rust)
gdb /path/to/binary core.dump
bt full   # Full backtrace
info locals

Step 3: Isolate

Binary search strategy:

Codebase: [file1, file2, ..., fileN]
1. Comment out half the code (files A-M)
2. Does the bug still reproduce?
   YES → bug is in files A-M → split A-M
   NO  → bug is in files N-Z → split N-Z
3. Repeat until single line identified

Configuration isolation:

# Compare configs
diff production.yml local.yml

# Start with minimal config
cp minimal.yml config.yml
# Add config values one at a time until bug appears

Dependency isolation:

# Check dependency versions
npm list
# Upgrade/downgrade suspect dependency
npm install package@version

Step 4: Use Debugging Tools

// TypeScript/JavaScript: Chrome DevTools or VS Code debugger
// 1. Set breakpoint before the crash
// 2. Step through to find null/undefined value
// 3. Check call stack for unexpected caller

// Node.js --inspect
node --inspect-brk app.js
// Open chrome://inspect in Chrome

// Conditional logging (don't leave in production)
console.log('[DEBUG] user:', user?.id, 'data:', JSON.stringify(data, null, 2));

# Python: pdb / ipdb
import pdb; pdb.set_trace()  # Python 3.6-
breakpoint()  # Python 3.7+

# Logging
import logging
logging.basicConfig(level=logging.DEBUG)
logger.debug(f"Processing item {item_id}, state: {state}")

// Rust: dbg! macro
let result = dbg!(some_expression());
// Prints: file.rs:42: some_expression() = <value>

// C#: Debugger.Launch + logging
System.Diagnostics.Debugger.Launch();
Console.WriteLine($"Processing: {item.Id}, State: {item.State}");

Step 5: Fix and Verify

fix:
  root_cause: "Null reference in UserService.GetName() when user has no profile"
  fix: "Add null check before accessing profile.Name"
  verification:
    - "Unit test added for null profile case"
    - "Existing tests still pass"
    - "Manually reproduce scenario → no crash"
  prevention:
    - "Add nullable reference types (C#) / strictNullChecks (TypeScript)"
    - "Add contract test for GetName()"
    - "Consider using Option/Maybe type instead of null"

Common Pitfalls

Best Practices

Advanced Techniques

Debugging Race Conditions

// Add thread-safe logging with timestamps
const debug = (msg: string) =>
  console.log(`[${Date.now()}] [${process.pid}] ${msg}`);

// Use happens-before tracking
let eventOrder: string[] = [];
async function track(step: string) {
  eventOrder.push(`${Date.now()}: ${step}`);
}

Debugging Memory Leaks

# Node.js heap dump
node --heapsnapshot-signal SIGUSR2 app.js
kill -USR2 <pid>  # Generates heap snapshot

# Chrome: Memory tab → Take heap snapshot
# Look for: detached DOM nodes, growing arrays, event listeners

Debugging Production (without SSH)

techniques:
  - Feature flags to enable debug logging remotely
  - Structured logging (JSON) to centralized log system
  - Distributed tracing (OpenTelemetry)
  - Crash reporting (Sentry, Bugsnag, AppSignal)
  - Session replay (FullStory, LogRocket, Hotjar)
  - Health check endpoints (/health, /debug/vars)
  - Metrics-based debugging (Grafana dashboard per service)

References

• references/debugging-strategy-advanced.md — Debugging Strategy Advanced Topics
• references/debugging-strategy-fundamentals.md — Debugging Strategy Fundamentals
• references/debugging-tools.md — Debugging Tools Reference
• references/root-cause-analysis.md — Root Cause Analysis Reference

Handoff

Hand off to dev-loop-performance-profiler if the bug is performance-related. Hand off to dev-loop-code-review for security-related bugs.

Implementation Patterns

Bug Report Parser

from typing import Dict, Optional, List
import re
from datetime import datetime

class BugReport:
    def __init__(self, title: str, description: str):
        self.title = title
        self.description = description
        self.severity = self._classify_severity()
        self.category = self._classify_category()
        self.reproducibility = self._assess_reproducibility()

    def _classify_severity(self) -> str:
        critical_kw = ["crash", "segfault", "data loss", "security", "0-day",
                       "production down", "p0", "p1", "critical"]
        high_kw = ["incorrect", "wrong", "broken", "not working", "fails",
                   "error", "exception", "regression"]
        text = f"{self.title} {self.description}".lower()
        for kw in critical_kw:
            if kw in text:
                return "critical"
        for kw in high_kw:
            if kw in text:
                return "high"
        return "medium"

    def _classify_category(self) -> str:
        text = f"{self.title} {self.description}".lower()
        if any(w in text for w in ["race", "async", "concurrent", "thread", "timing"]):
            return "race_condition"
        if any(w in text for w in ["null", "undefined", "non-null", "optional"]):
            return "null_reference"
        if any(w in text for w in ["memory", "leak", "oom", "heap"]):
            return "memory"
        if any(w in text for w in ["perf", "slow", "timeout", "hang"]):
            return "performance"
        if any(w in text for w in ["api", "http", "response", "status", "400", "500"]):
            return "api_integration"
        if any(w in text for w in ["database", "query", "sql", "migration"]):
            return "database"
        return "logic"

    def _assess_reproducibility(self) -> str:
        text = f"{self.title} {self.description}".lower()
        if any(w in text for w in ["always", "every time", "100%", "consistently"]):
            return "always"
        if any(w in text for w in ["sometimes", "intermittent", "occasionally", "random"]):
            return "intermittent"
        if any(w in text for w in ["once", "single time", "one time", "rarely"]):
            return "rare"
        return "unknown"

Root Cause Analysis Documenter

from typing import List, Dict, Optional
from datetime import datetime
import json

class RCADocument:
    def __init__(self, bug_id: str, title: str):
        self.bug_id = bug_id
        self.title = title
        self.created_at = datetime.utcnow()
        self.timeline: List[Dict] = []
        self.hypotheses: List[Dict] = []
        self.root_cause: Optional[Dict] = None
        self.fix: Optional[Dict] = None

    def add_hypothesis(self, description: str, probability: float, evidence: List[str]):
        self.hypotheses.append({
            "description": description,
            "probability": probability,
            "evidence": evidence,
            "status": "proposed",
            "test_result": None,
        })

    def record_test(self, hypothesis_index: int, confirmed: bool, notes: str):
        if hypothesis_index < len(self.hypotheses):
            self.hypotheses[hypothesis_index]["status"] = "confirmed" if confirmed else "rejected"
            self.hypotheses[hypothesis_index]["test_result"] = notes

    def set_root_cause(self, description: str, file: str, line: int, category: str):
        self.root_cause = {
            "description": description,
            "file": file,
            "line": line,
            "category": category,
            "discovered_at": datetime.utcnow().isoformat(),
        }

    def set_fix(self, description: str, files_changed: List[str], prevention: List[str]):
        self.fix = {
            "description": description,
            "files_changed": files_changed,
            "prevention": prevention,
            "applied_at": datetime.utcnow().isoformat(),
        }

    def get_5_whys(self) -> List[str]:
        if not self.root_cause:
            return ["Root cause not identified"]
        whys = []
        cause = self.root_cause["description"]
        why_count = 0
        while cause and why_count < 5:
            whys.append(f"Why? {cause}")
            cause = self._get_deeper_cause(cause)
            why_count += 1
        return whys

    def _get_deeper_cause(self, cause: str) -> Optional[str]:
        deeper = {
            "null check missing": "developer didn't consider edge case",
            "no input validation": "trusted external input without sanitization",
            "wrong return type": "interface contract not documented",
            "race condition": "shared state without synchronization",
        }
        cause_lower = cause.lower()
        for key, val in deeper.items():
            if key in cause_lower:
                return val
        return None

    def summary(self) -> str:
        return json.dumps({
            "bug_id": self.bug_id,
            "title": self.title,
            "severity": "high" if self.root_cause else "unknown",
            "root_cause": self.root_cause["description"] if self.root_cause else "pending",
            "hypotheses_tested": sum(1 for h in self.hypotheses if h["status"] != "proposed"),
            "prevention": self.fix["prevention"] if self.fix else [],
        }, indent=2)

Git Bisect Automation

# Automated git bisect script
cat > /tmp/bisect.sh << 'SCRIPT'
#!/bin/bash
# Usage: git bisect run /tmp/bisect.sh

# Build the project
npm run build > /dev/null 2>&1

# Run the specific test that catches the bug
npx jest --testPathPattern="tests/specific-bug-test" 2>&1 | grep -q "FAIL"

# Exit 0 if good (test passes), 1 if bad (test fails)
# Git bisect expects 0 for good, 1 for bad (or 125 for skip)
SCRIPT
chmod +x /tmp/bisect.sh

# Start bisect
git bisect start HEAD v1.0.0
git bisect run /tmp/bisect.sh

Architecture Decision Trees

Debugging Tool Selection

What language/platform?
├── JavaScript / TypeScript
│   ├── Node.js backend → --inspect + Chrome DevTools / VS Code
│   ├── Browser → Chrome DevTools / React DevTools / Redux DevTools
│   └── Mobile (React Native) → Flipper / React Native Debugger
│
├── Python
│   ├── Local → pdb / ipdb / breakpoint()
│   └── Production → traceback + structured logging + Sentry
│
├── Java / JVM
│   ├── Local → IntelliJ debugger / JDB / VisualVM
│   └── Production → JMX / JFR / heap dump / thread dump
│
├── .NET / C#
│   ├── Local → VS / Rider debugger / dotnet-trace
│   └── Production → dotnet-counters / dotnet-dump
│
├── Rust
│   ├── Local → lldb / gdb / rust-gdb
│   └── Panic → RUST_BACKTRACE=1 / RUST_LIB_BACKTRACE=1
│
└── Go
    ├── Local → delve debugger
    └── Production → pprof / trace

Hypothesis Prioritization

Given multiple possible causes, which to test first?
├── Most recent change (git log, deployment)
├── Most likely based on error message
├── Easiest to test (quick experiment)
├── Most common cause for this type of bug
└── Components involved in critical path

Production Considerations

• Structured error logging: Log all errors with correlation IDs, stack traces, and context. Use a consistent JSON format consumable by log aggregation tools.
• Automatic error triage: Set up automated error grouping by stack trace fingerprint. Route to appropriate team based on affected service.
• Canary debugging: Enable verbose debug logging for specific users or sessions via feature flags. Avoids deploying debug builds to all users.
• Production breakpoints: Use tools like Lightrun or Rookout for on-demand production debugging without redeployment or restart.

Anti-Patterns

Performance Optimization

• Binary search on code: Use git bisect for regression bugs. Automates the most efficient search strategy — O(log N) commits checked instead of O(N).
• Focused test for reproduction: Write a minimal test case that reproduces the bug. Avoids running full test suite during fix verification.
• Conditional breakpoints: Use data-dependent breakpoints to stop only when relevant conditions are met. Avoids manual stepping through thousands of iterations.