Debugging Techniques Advanced

Advanced Debugging Techniques

Beyond print-debugging — reverse debugging, time-travel, core dump analysis, tracing, and systematic root cause analysis.

When to Use

• Bugs that are hard to reproduce consistently (race conditions, heisenbugs)
• Crashes that only occur in production
• Performance regressions needing profiling
• Memory corruption or undefined behavior
• Complex multi-threaded or distributed system bugs

Systematic Root Cause Analysis

The Scientific Method

def debug_systematically(failure_scenario):
    """
    1. HYPOTHESIS: Form a hypothesis about the root cause
    2. PREDICTION: Predict what will happen if hypothesis is true
    3. EXPERIMENT: Design minimal experiment to test
    4. OBSERVE: Run and collect data
    5. CONFIRM/REFUTE: Does evidence support or reject hypothesis?
    """
    
    techniques = [
        'reproduce_minimally',    # Simplest input that triggers bug
        'binary_search_history',  # git bisect to find regression
        'add_assertions',         # Contract checking
        'enable_logging',         # Targeted log points
        'check_recent_changes',   # What changed near regression time
        'isolate_component',      # Test component in isolation
    ]

Time-Travel Debugging (rr)

# rr — record and replay execution (Linux)
# Record: captures all non-determinism
rr record ./my_program --arg1 value

# Replay
rr replay

# Reverse-continue in gdb within rr
# reverse-continue  — go back to previous crash
# reverse-step      — step backward
# reverse-next      — step over backward

# Watchpoint + reverse-continue to find when variable changed
watch -l my_variable
reverse-continue
# Lands exactly on the instruction that modified the variable

# Randomize thread scheduling to surface race conditions
rr record --chaos

Core Dump Analysis

# Enable core dumps
ulimit -c unlimited
echo "/tmp/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern

# Analyze with gdb
gdb ./my_program /tmp/core.my_program.12345

# In gdb:
# bt              — backtrace (call stack)
# bt full         — backtrace with local variables
# frame 3         — switch to frame 3
# info locals     — show local variables
# p variable      — print variable

Python Advanced Debugging

Post-Mortem on Exception

import sys, traceback, pdb

def post_mortem_on_error():
    """Enter pdb on any unhandled exception."""
    def excepthook(type, value, tb):
        traceback.print_exception(type, value, tb)
        pdb.post_mortem(tb)
    sys.excepthook = excepthook

Memory Leak Detection

import tracemalloc
import objgraph

class MemoryDebugger:
    @staticmethod
    def start(): tracemalloc.start()
    
    @staticmethod
    def show_top(n=10):
        snapshot = tracemalloc.take_snapshot()
        for stat in snapshot.statistics('lineno')[:n]:
            print(f"{stat.count:6d} × {stat.size_kb:8.1f} KB")
    
    @staticmethod
    def show_growth():
        """Show memory growth since last call."""
        current = tracemalloc.take_snapshot()
        if hasattr(MemoryDebugger, '_last'):
            for d in current.compare_to(MemoryDebugger._last, 'lineno')[:10]:
                print(f"{d.size_diff:8.1f} KB {d.count_diff:4d}")
        MemoryDebugger._last = current

Rust Debugging

// Set for full backtraces
// RUST_BACKTRACE=1
// RUST_LIB_BACKTRACE=1

// dbg! macro — prints file, line, and value
fn example() {
    let x = 42;
    let y = dbg!(x * 2);  // [src/main.rs:4] x * 2 = 84
}

Common Pitfalls

1. Heisenbugs — logging/printing changes timing; use rr for time-sensitive bugs
2. Not minimizing the repro case — debugging in full system is slow; isolate
3. Confirmation bias — actively try to disprove your hypothesis
4. Skipping binary search — git bisect finds the offending commit in O(log n) steps
5. No core dumps — a crash without a dump is lost information; enable them
6. Production debug overhead — don't run heavy debuggers in prod; use post-mortem

Verification Checklist

• Reproducer minimized to simplest possible input
• Git bisect identified the regression commit
• Core dumps enabled and retrievable in production
• Root cause confirmed (changing the cause changes the symptom)
• Fix verified by tests
• Regression test added

See Also

• systematic-debugging — structured 4-phase debugging
• advanced-version-control-git — using bisect
• performance-optimization — profiling
• node-inspect-debugger — Node.js debugging