Systematic Debugging

Systematic Debugging — amd-smi

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find the root cause before attempting a fix. Symptom fixes are failure.

Violating the letter of this process is violating the spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT-CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose a fix.

When to Use

Use for any technical issue:

• Test failures (C++ GTest, Python unit, CLI)
• Bugs reported in amdsmi_* output
• Unexpected CLI output
• Build/CMake failures
• Packaging failures (RPM/DEB postinst)
• Wrapper-regen mismatches
• Performance regressions

Especially when:

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• You've already tried 2+ fixes
• The previous fix didn't work
• You don't fully understand the issue

The Four Phases

Complete each phase before proceeding to the next.

Phase 1: Root-Cause Investigation

1. Read error messages carefully

   • Read the full stack trace, every frame
   • Note exact line numbers, file paths, error codes
   • For amdsmi_status_t errors: which enum value? Where is it set?

2. Reproduce consistently

   • Exact command, exact environment
   • Same hardware? Same install context (system vs pip)?
   • If not reproducible → gather more data, don't guess

3. Check recent changes

   • git log --oneline -20 -- <affected files>
   • git diff HEAD~5 -- <affected file>
   • New CMake options? New dependency? Wrapper regenerated?

4. Gather evidence across cascade layers

   amd-smi bugs often span layers. Add instrumentation at every boundary:

   # Layer 1: CLI argparse
   echo "argparse args: $@"
   
   # Layer 2: Python interface
   python3 -c "import amdsmi; print(amdsmi.amdsmi_get_lib_version())"
   
   # Layer 3: C wrapper loaded
   python3 -c "from amdsmi import amdsmi_wrapper; print(amdsmi_wrapper.libamd_smi._name)"
   
   # Layer 4: Actual C call return value
   strace -e trace=openat python3 -c "import amdsmi; amdsmi.amdsmi_init()" 2>&1 | grep libamd_smi
   

   This reveals which layer breaks, not just that something breaks.

5. Trace data flow backward

   • Where does the bad value originate?
   • What called this with the bad value?
   • Keep tracing up the cascade until you find the source
   • Fix at the source, not at the symptom

Phase 2: Pattern Analysis

1. Find a working example — locate the closest-working similar function/command/test
2. Compare against the reference — read it completely, line by line; don't skim
3. List every difference — however small. Don't assume "that can't matter"
4. Understand dependencies — what config, environment, build flags does the working version need?

Phase 3: Hypothesis and Testing

1. Form a single hypothesis — "I think X is the root cause because Y." Write it down.
2. Test minimally — the smallest possible change. One variable at a time. No "while I'm here" fixes.
3. Verify before continuing — worked? Phase 4. Didn't? Form a NEW hypothesis. Don't pile fixes.
4. When you don't know — say "I don't understand X". Don't pretend. Ask the user or research more.

Phase 4: Implementation

1. Create a failing test that reproduces the bug

   • Use the test-driven-development skill
   • The test MUST fail before the fix and pass after

2. Implement a single fix — address the root cause. One change. No bundled refactors.

3. Verify

   • Bug test passes
   • No other tests broken
   • Issue is actually resolved end-to-end (use verification-before-completion)

4. If the fix doesn't work — STOP. Count attempts.

   • < 3 attempts → return to Phase 1 with the new information
   • ≥ 3 attempts → STOP and question the architecture (next item)

5. If 3+ fixes failed: question the architecture

   Pattern indicating an architectural problem:

   • Each fix reveals new shared state / coupling in a different place
   • Fixes require "massive refactoring" to implement
   • Each fix creates new symptoms elsewhere

   Stop. Discuss with the user. This is not a failed hypothesis — this is a wrong architecture.

Red Flags — STOP and Follow Process

• "Quick fix for now, investigate later"
• "Just try changing X and see if it works"
• "Skip the test, I'll manually verify"
• "It's probably X, let me fix that"
• "I don't fully understand but this might work"
• "Pattern says X but I'll adapt it differently"
• Proposing solutions before tracing data flow
• "One more fix attempt" after 2+ failures
• Each fix reveals a new problem in a different place

All of these mean: stop, return to Phase 1.

If 3+ fixes failed: question the architecture (Phase 4 step 5).

Common Rationalizations

Quick Reference

amd-smi-Specific Investigation Tips

• API cascade bugs: Use the cascade Quick Check (grep across header + cc + wrapper + interface + CLI) — gaps are often the root cause
• Wrapper mismatch: If a function's Python signature differs from the C header, run tools/update_wrapper.sh — the wrapper drifted
• Install context bugs: Reproduce in BOTH system install and pip install before claiming the fix works (see amdsmi-build-install skill)
• CMake bugs: Always rm -rf build between attempts — stale CMake cache hides real causes
• Test flakiness: Run the test in isolation, then alone in the suite, then with the full suite — find the polluting test

Required Companion Skills

• test-driven-development — Phase 4 Step 1 (failing test)
• verification-before-completion — Phase 4 Step 3 (verify the fix)
• dispatching-parallel-agents — when there are 2+ independent failures