Root Cause Tracing

Core rule: Never fix just where error appears. Fix where problem originates.

Example symptoms:

• "Database rejects empty string" ← Where did empty string come from?
• "File not found: ''" ← Why is path empty?
• "Invalid argument to function" ← Who passed invalid argument?
• "Null pointer dereference" ← What should have been initialized?

Read the complete error:

Error: Invalid email format: ""
  at validateEmail (validator.ts:42)
  at UserService.create (user-service.ts:18)
  at ApiHandler.createUser (api-handler.ts:67)
  at HttpServer.handleRequest (server.ts:123)
  at TestCase.test_create_user (user.test.ts:10)

Symptom: Email validation fails on empty string Location: Deep in validator utility

DON'T fix here yet. This might be symptom, not source.

2. Find Immediate Cause

What code directly causes this?

// validator.ts:42
function validateEmail(email: string): boolean {
  if (!email) throw new Error(`Invalid email format: "${email}"`);
  return EMAIL_REGEX.test(email);
}

Question: Why is email empty? Keep tracing.

3. Trace Backward: What Called This?

Use stack trace:

// user-service.ts:18
create(request: UserRequest): User {
  validateEmail(request.email); // Called with request.email = ""
  // ...
}

Question: Why is request.email empty? Keep tracing.

4. Keep Tracing Up the Stack

// api-handler.ts:67
async createUser(req: Request): Promise<Response> {
  const userRequest = {
    name: req.body.name,
    email: req.body.email || "", // ← FOUND IT!
  };
  return this.userService.create(userRequest);
}

Root cause found: API handler provides default empty string when email missing.

5. Identify the Pattern

Why empty string as default?

• Misguided "safety": Thought empty string better than undefined
• Should reject invalid request at API boundary
• Downstream code assumes data already validated

6. Fix at Source

// api-handler.ts (SOURCE FIX)
async createUser(req: Request): Promise<Response> {
  if (!req.body.email) {
    return Response.badRequest("Email is required");
  }
  const userRequest = {
    name: req.body.name,
    email: req.body.email, // No default, already validated
  };
  return this.userService.create(userRequest);
}

7. Add Defense in Depth

After fixing source, add validation at each layer as backup:

// Layer 1: API - Reject invalid input (PRIMARY FIX)
if (!req.body.email) return Response.badRequest("Email required");

// Layer 2: Service - Validate assumptions
assert(request.email, "email must be present");

// Layer 3: Utility - Defensive check
if (!email) throw new Error("invariant violated: email empty");

Primary fix at source. Defense is backup, not replacement.

IMPORTANT: Claude cannot run interactive debuggers. Guide user through debugger commands.

"Let's use lldb to trace backward through the call stack.

Please run these commands:
  lldb target/debug/myapp
  (lldb) breakpoint set --file validator.rs --line 42
  (lldb) run

When breakpoint hits:
  (lldb) frame variable email     # Check value here
  (lldb) bt                       # See full call stack
  (lldb) up                       # Move to caller
  (lldb) frame variable request   # Check values in caller
  (lldb) up                       # Move up again
  (lldb) frame variable           # Where empty string created?

Please share:
  1. Value of 'email' at validator.rs:42
  2. Value of 'request.email' in user_service.rs
  3. Value of 'req.body.email' in api_handler.rs
  4. Where does empty string first appear?"

Option 2: Add Instrumentation (Claude CAN Do This)

When debugger not available or issue intermittent:

// Add at error location
fn validate_email(email: &str) -> Result<()> {
    eprintln!("DEBUG validate_email called:");
    eprintln!("  email: {:?}", email);
    eprintln!("  backtrace: {}", std::backtrace::Backtrace::capture());

    if email.is_empty() {
        return Err(Error::InvalidEmail);
    }
    // ...
}

Critical: Use eprintln!() or console.error() in tests (not logger - may be suppressed).

Run and analyze:

cargo test 2>&1 | grep "DEBUG validate_email" -A 10

Look for:

• Test file names in backtraces
• Line numbers triggering the call
• Patterns (same test? same parameter?)

When something appears during tests but you don't know which:

Binary search approach:

# Run half the tests
npm test tests/first-half/*.test.ts
# Pollution appears? Yes → in first half, No → second half

# Subdivide
npm test tests/first-quarter/*.test.ts

# Continue until specific file
npm test tests/auth/login.test.ts  ← Found it!

Or test isolation:

# Run tests one at a time
for test in tests/**/*.test.ts; do
  echo "Testing: $test"
  npm test "$test"
  if [ -d .git ]; then
    echo "FOUND POLLUTER: $test"
    break
  fi
done

Error: "Invalid argument: empty directory"

Developer adds validation at symptom:

fn git_init(directory: &str) { if directory.is_empty() { panic!("Directory cannot be empty"); // Band-aid } Command::new("git").arg("init").current_dir(directory).run() }

1. git_init called with directory=""
2. WorkspaceManager.init(projectDir="")
3. Session.create(projectDir="")
4. Test: Project.create(context.tempDir)
5. SOURCE: context.tempDir="" (accessed before beforeEach!)

Fix at source:

function setupTest() {
  let _tempDir: string | undefined;

  return {
    beforeEach() {
      _tempDir = makeTempDir();
    },
    get tempDir(): string {
      if (!_tempDir) {
        throw new Error("tempDir accessed before beforeEach!");
      }
      return _tempDir;
    }
  };
}

What you gain:

• Fixes actual bug (test timing issue)
• Prevents same mistake elsewhere
• Clear error at source, not deep in stack
• No empty strings propagating through system

Developer sees empty string default and "fixes" it:

email: req.body.email || "noreply@example.com"

Ships to production

Bug: Users created without email input get noreply@example.com

Database has fake emails, can't distinguish missing from real

1. Why was empty string default?
2. Should email be optional or required?
3. What does API spec say?
4. What does database schema say?

Findings:

• Email column is NOT NULL in database
• API docs say email is required
• Empty string was workaround, not design

Fix at source (validate at boundary):

async createUser(req: Request): Promise<Response> {
  // Validate at API boundary
  if (!req.body.email) {
    return Response.badRequest("Email is required");
  }

  const userRequest = {
    name: req.body.name,
    email: req.body.email, // No default needed
  };
  return this.userService.create(userRequest);
}

What you gain:

• Validates at correct layer (API boundary)
• Clear error message to client
• No invalid data propagates downstream
• Database constraints enforced
• Matches API specification

Symptom location:

Error: Cannot initialize git repo (repo already exists) Location: src/workspace/git.rs:45

Developer adds check:

if Path::new(".git").exists() { return Err("Git already initialized"); }

Doesn't help - still appears in wrong place!

1. git init runs with cwd=""
   ↓ Why is cwd empty?

2. WorkspaceManager.init(projectDir="")
   ↓ Why is projectDir empty?

3. Session.create(projectDir="")
   ↓ Why was empty string passed?

4. Test: Project.create(context.tempDir)
   ↓ Why is context.tempDir empty?

5. ROOT CAUSE:
   const context = setupTest(); // tempDir="" initially
   Project.create(context.tempDir); // Accessed at top level!

   beforeEach(() => {
     context.tempDir = makeTempDir(); // Assigned here
   });

   TEST ACCESSED TEMPDIR BEFORE BEFOREEACH RAN!

Fix at source (make early access impossible):

function setupTest() {
  let _tempDir: string | undefined;

  return {
    beforeEach() {
      _tempDir = makeTempDir();
    },
    get tempDir(): string {
      if (!_tempDir) {
        throw new Error("tempDir accessed before beforeEach!");
      }
      return _tempDir;
    }
  };
}

Then add defense at each layer:

// Layer 1: Test framework (PRIMARY FIX)
// Getter throws if accessed early

// Layer 2: Project validation
fn create(directory: &str) -> Result<Self> {
    if directory.is_empty() {
        return Err("Directory cannot be empty");
    }
    // ...
}

// Layer 3: Workspace validation
fn init(path: &Path) -> Result<()> {
    if !path.exists() {
        return Err("Path must exist");
    }
    // ...
}

// Layer 4: Environment guard
fn git_init(dir: &Path) -> Result<()> {
    if env::var("NODE_ENV") != Ok("test".to_string()) {
        if !dir.starts_with("/tmp") {
            panic!("Refusing to git init outside test dir");
        }
    }
    // ...
}

What you gain:

• Primary fix prevents early access (source)
• Each layer validates assumptions (defense)
• Clear error at source, not deep in stack
• Environment guard prevents production pollution
• Multi-layer defense catches future mistakes

1. Never fix just where error appears → Trace backward to find source
2. Don't stop at first suspicious code → Keep tracing to original trigger
3. Fix at source first → Defense is backup, not primary fix
4. Use debugger OR instrumentation → Don't guess at call chain
5. Add defense at each layer → After fixing source, validate assumptions throughout

Common Excuses

All of these mean: STOP. Trace backward to find source.

• "Error is obvious here, I'll add validation" (That's a symptom fix)
• "Stack trace shows the problem" (Shows symptom location, not source)
• "This code should handle empty values" (Why is value empty? Find source.)
• "Too deep to trace, I'll add defensive check" (Defense without source fix = band-aid)
• "Multiple places could cause this" (Trace to find which one actually does)

• Traced backward through entire call chain
• Found where invalid data was created (not just passed)
• Identified WHY invalid data was created (pattern/assumption)
• Fixed at source (where bad data originates)
• Added defense at each layer (validate assumptions)
• Verified fix with test (reproduces original bug, passes with fix)
• Confirmed no other code paths have same pattern

Can't check all boxes? Keep tracing backward.

This skill requires:

• Stack traces or debugger access
• Ability to add instrumentation (logging)
• Understanding of call chain

This skill calls:

• hyperpowers:test-driven-development (write regression test after finding source)
• hyperpowers:verification-before-completion (verify fix works)

When stuck:

• Can't find source → Add instrumentation at each layer, run test
• Stack trace unclear → Use debugger to inspect variables at each frame
• Multiple suspects → Add instrumentation to all, find which actually executes
• Intermittent issue → Add instrumentation and wait for reproduction