Debugging

Debugging Protocol (v2 -- Friction-Hardened)

Built from 169 real sessions. Addresses 38 "wrong approach" events, 30 "buggy code" events, 17 "misunderstood request" events. Every rule below exists because it was violated in production.

When This Skill Activates

• "Debug this", "fix this bug", "not working"
• Error messages, stack traces
• "Undefined", "null", "crash", "fails"
• Unexpected behavior
• Performance issues
• Database query returning wrong data
• "Who changed X?" / audit trail questions

STOP-THINK-ACT Protocol (MANDATORY -- Execute Before EVERY Action)

Origin: 38 sessions had "wrong approach" friction -- Claude jumped to the wrong tool, wrong file, or wrong strategy before understanding context. This protocol is non-negotiable.

STOP -- Before writing code, running commands, or choosing an approach:

THINK -- Before stating conclusions:

ACT -- Only after confirming context and approach:

1. Make the minimal change needed. Don't refactor surrounding code.
2. Verify the change works before declaring done. Show proof.
3. If something fails, diagnose root cause -- don't retry the same failing approach.
4. If unsure, ask the user rather than guessing.

Anti-Hallucination Rules (NEVER violate)

Multi-System Identification

When debugging involves multiple systems or databases, always confirm which system the user is referring to before querying. Never assume.

Identify the System First

User mentions a bug/issue
        |
        v
Does the user specify a system?
        |
   YES -+-> Use that system
        |
   NO --+-> Look for contextual clues in their message
             |
             +-- Database-related keywords? --> Confirm which database
             +-- Server/infrastructure keywords? --> Confirm which server
             +-- None of the above or ambiguous
                 --> ASK: "Which system should I investigate?"

Schema Verification (MANDATORY)

BEFORE writing any query:
  1. Identify target table(s)
  2. Run schema check (SELECT TOP 1 *, sp_columns, or MCP metadata tools)
  3. Verify column names MATCH what you plan to use
  4. Only THEN write the actual query

NEVER skip this step. NEVER assume column names.

Anti-Exploration-Spiral Rule

Origin: Claude spent entire sessions exploring without executing fixes. Multiple sessions consumed 100% of context just reading files without ever fixing anything.

The Rule

INVESTIGATION BUDGET:
  - Time-box investigation to 3-5 tool calls before forming a hypothesis
  - After 5 exploration calls without a hypothesis: STOP and ask the user
  - After 8 exploration calls total: You MUST have started fixing OR explained to user why not
  - NEVER spend an entire session only exploring/planning when user asked for a fix

Investigation Checkpoints

Self-Check Questions (Ask Yourself Every 3 Calls)

1. "Do I have enough information to form a hypothesis?" -- If yes, state it and fix.
2. "Am I reading files that are actually relevant, or am I spiraling?" -- If spiraling, stop.
3. "Did the user ask for exploration or a fix?" -- If fix, execute.
4. "Am I repeating the same type of investigation?" -- If yes, try a different angle or ask.

Evidence-Before-Conclusions Enforcement

Origin: 17 "misunderstood request" events. Claude attributed root causes without proof, blamed team members without evidence, and stated conclusions before showing data.

The Evidence Protocol

WRONG ORDER (violation):
  1. "The bug is caused by X"          <-- conclusion first
  2. "Let me verify..." (maybe)        <-- evidence after
  3. User has to correct you           <-- friction

RIGHT ORDER (required):
  1. "Let me gather evidence..."       <-- investigate first
  2. "Here's what I found: [data]"     <-- show raw evidence
  3. "Based on this, the cause is X"   <-- evidence-backed conclusion

Attribution Rules

When investigating "who changed X" or "what caused this":

NEVER DO:
  "This was probably caused by [person/event]"
  "Someone must have changed..."
  "This looks like it was broken by..."

ALWAYS DO:
  1. Run: git log --oneline -20 -- path/to/file
  2. Run: git blame path/to/file | grep "relevant_line"
  3. Run: git diff <commit_before>..<commit_after> -- path/to/file
  4. SHOW the diff/log output
  5. THEN say: "Commit [hash] by [author] on [date] changed [specific line]. Here's the diff: ..."

Showing Evidence (Templates)

For code bugs:

Evidence gathered:
- File: [path], line [N]
- Current value: [what the code does]
- Expected value: [what it should do]
- Proof: [test output / error log / stack trace]

Conclusion: [root cause based on above evidence]

For database issues:

Evidence gathered:
- Query: [the query run]
- Result: [actual output, first N rows]
- Expected: [what should have returned]
- Schema check: [confirmed columns exist]

Conclusion: [root cause based on above evidence]

For "who changed this" questions:

Evidence gathered:
- git blame output: [relevant lines]
- Commit: [hash], Author: [name], Date: [date]
- Diff: [what changed]
- Context: [commit message, PR if available]

Conclusion: [factual statement based on git data]

Systematic Debugging Process

Phase 0: STOP-THINK-ACT (MANDATORY -- Do This First)

Before ANY debugging action, execute the full STOP-THINK-ACT protocol above.

Minimum required confirmations before proceeding:

[ ] System/environment identified (which app? which server? which database?)
[ ] Output format confirmed (SQL? Code fix? Plan? Report?)
[ ] Target file/path verified (correct file, correct server, correct branch)
[ ] Database connection identified (if applicable)
[ ] User's actual request understood (fix? investigate? explain? plan?)

If ANY of these are unclear, ASK before proceeding.

Phase 1: Reproduce

Before fixing anything:

1. Can I reproduce the bug consistently?
2. What are the exact steps to trigger it?
3. What is the expected behavior?
4. What is the actual behavior?
5. What error message/log output do I see? (Show it BEFORE interpreting)

Phase 2: Isolate

Narrow down the problem (budget: 3-5 tool calls):

1. When did it last work? (check git history)
2. What changed recently? (git diff, git log)
3. Is it specific to certain inputs?
4. Does it happen in all environments?

Phase 3: Investigate

Gather concrete evidence (stay within investigation budget):

Read the Error Message Carefully

Error: Cannot read property 'map' of undefined
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       This tells you: something is undefined that should be an array

at UserList.render (UserList.tsx:15:23)
   ^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^
   Component name    File and line number

Key Investigation Commands

# Find where variable is defined
grep -r "variableName" --include="*.ts" -n

# Find where function is called
grep -r "functionName(" --include="*.ts" -n

# Check git history for recent changes
git log --oneline -20
git diff HEAD~5 -- path/to/file

# Find related tests
find . -name "*.test.ts" | xargs grep "FunctionName"

# Attribution investigation (when asked "who changed X")
git blame path/to/file
git log --oneline --follow -- path/to/file
git diff <old_commit>..<new_commit> -- path/to/file

For Database Issues (MANDATORY schema check)

-- STEP 1: ALWAYS check schema before querying
SELECT TOP 1 * FROM [table_name];
-- NEVER assume column names exist

-- STEP 2: Verify the table exists
SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 'xyz'

-- STEP 3: Only THEN write your actual diagnostic query
-- Match column names EXACTLY to what schema returned

Database anti-patterns to avoid:

• Writing a query with assumed column names (ALWAYS schema-check first)
• Querying the wrong database connection when multiple are available
• Forgetting required filters (tenant, site, company ID, etc.)
• Confusing snapshot/cached data with live data

Phase 4: Hypothesize

You MUST have a hypothesis by tool call 5. State it explicitly.

Form specific, evidence-backed hypotheses:

NOT: "Something is wrong with the data"
YES: "The API returns null when user has no orders, but we expect an empty array"

NOT: "User X probably changed this"
YES: "git blame shows line 45 was changed in commit abc123 on Jan 5 by [user]"

NOT: "The database might be wrong"
YES: "Column TRANSDATE contains UTC timestamps but the query filters by SGT without conversion"

Show your evidence to the user before proposing causes.

Phase 5: Test Hypothesis

Add strategic logging or run targeted queries:

// Before the error
console.log('DEBUG: data before map:', JSON.stringify(data, null, 2));
console.log('DEBUG: data type:', typeof data, Array.isArray(data));

Phase 6: Fix

Apply minimal, targeted fix:

// BAD: Overly broad fix
const items = data?.items?.map?.(x => x) ?? [];

// GOOD: Fix the actual problem at its source
if (!data.items) {
  throw new Error('Expected items array but got: ' + typeof data.items);
}

After fixing:

1. Verify the fix works (run test, check output, show proof)
2. Check for secondary config files or related files that may also need changes
3. Run existing tests to check for regressions
4. Show the user proof the fix works before declaring done

Phase 7: Verify & Report

REQUIRED before declaring "done":
  1. Show the fix applied (diff or code snippet)
  2. Show proof it works (test output, query result, application behavior)
  3. Show no regressions (existing tests still pass)
  4. If database fix: re-run the original failing query to show correct results

Parallel Agent Investigation Pattern

For multi-system issues (e.g., data mismatch between frontend and backend), dispatch parallel sub-agents to investigate different systems simultaneously.

When to Use Parallel Investigation

• Bug involves data flowing between 2+ systems (API -> Database -> Report)
• Need to compare state across environments (prod vs staging)
• Root cause could be in multiple places (frontend AND backend AND database)
• Time pressure -- need to investigate multiple angles at once

Pattern: Coordinator + Investigators

COORDINATOR (you):
  1. Identify which systems/areas need investigation
  2. Dispatch sub-agents with SPECIFIC investigation tasks
  3. Each sub-agent gets:
     - Exact system to investigate
     - Specific questions to answer
     - Investigation budget (max tool calls)
     - Required evidence format
  4. Synthesize findings from all sub-agents
  5. Form unified hypothesis based on all evidence

SUB-AGENT TEMPLATE:
  "Investigate [SYSTEM] for [ISSUE].
   Check: [specific things to look for]
   Return: [evidence format needed]
   Budget: [max N tool calls]
   Do NOT fix anything -- investigation only."

Example: Data Mismatch Between API and Report

Sub-agent 1 (Database):
  "Query the primary database for orders in date range X-Y.
   Check: Do the totals match the expected value?
   Return: Row count, total amount, sample of 5 rows."

Sub-agent 2 (Cache/Replica):
  "Query the read replica or cache for same date range.
   Check: Are all records present? Any replication lag?
   Return: Row count, total amount, latest sync timestamp."

Sub-agent 3 (Report):
  "Check the report logic/query.
   Check: Does it filter correctly? Any timezone issues?
   Return: The actual query used and any filter logic."

Coordinator synthesizes:
  "Primary DB has 150 records totaling $X. Replica has 148 -- missing 2 records
   due to replication lag. Report query is correct but reads from replica,
   so it's missing 2 records. Root cause: replication delay, not a bug."

Output Format Discipline

Origin: Claude delivered the wrong output format -- gave one language/format when user explicitly asked for another. 17 "misunderstood request" events.

Format Decision Matrix

Self-Check Before Responding

Before writing your response, verify:
  [ ] My output format matches what the user asked for
  [ ] If user said "SQL", I'm giving SQL (not something else)
  [ ] If user said "fix", I'm fixing (not just exploring/planning)
  [ ] If user said "plan", I'm planning (not coding)
  [ ] If ambiguous, I'm asking for clarification

Common Bug Patterns

1. Undefined/Null Access

// BUG
user.profile.name  // crashes if profile is undefined

// FIX
user?.profile?.name  // safe access
// OR
if (!user?.profile) throw new Error('User profile required');

2. Async/Await Issues

// BUG: Missing await
const data = fetchData();  // Returns Promise, not data
console.log(data.items);   // undefined

// FIX
const data = await fetchData();

3. State Race Conditions

// BUG: State not updated yet
setCount(count + 1);
console.log(count);  // Still old value

// FIX: Use callback form
setCount(prev => prev + 1);

4. Closure Stale Values

// BUG: Stale closure
useEffect(() => {
  const interval = setInterval(() => {
    console.log(count);  // Always logs initial value
  }, 1000);
}, []);  // Missing dependency

// FIX
useEffect(() => {
  const interval = setInterval(() => {
    console.log(count);
  }, 1000);
  return () => clearInterval(interval);
}, [count]);  // Add dependency

5. Off-by-One Errors

// BUG
for (let i = 0; i <= array.length; i++)  // Goes one too far

// FIX
for (let i = 0; i < array.length; i++)

6. Type Coercion

// BUG
if (value == 0)   // "" == 0 is true!
if (value == "")  // 0 == "" is true!

// FIX
if (value === 0)
if (value === "")

7. Wrong File / Wrong Environment

// BUG: Editing local file when user means remote VPS config
// BUG: Editing .env when the app reads from config.json
// BUG: Editing the right file on the wrong branch

// FIX: Always confirm file path and environment in Phase 0

8. Wrong Database / Wrong System

// BUG: Querying the wrong database when multiple are available
// BUG: Using column names from one system in queries for another
// BUG: Forgetting required filters (tenant ID, site ID, company code, etc.)

// FIX: Always confirm which system/database before querying

9. Timezone / Data Staleness

// BUG: Comparing snapshot data with live data without accounting for lag
// BUG: Filtering by local timezone when database stores UTC
// BUG: Missing records because they were created after snapshot cutoff

// FIX: Know your data source's refresh cycle and timezone

10. Silent Failure / Swallowed Errors

// BUG: Error swallowed, appears to "work" but doesn't
try {
  await saveData(data);
} catch (e) {
  // silently fails
}

// FIX: Handle or re-throw
try {
  await saveData(data);
} catch (e) {
  logger.error('Failed to save data:', e);
  throw e;  // Or handle meaningfully
}

Debugging Tools

Console Methods

console.log(variable);                    // Basic output
console.table(arrayOfObjects);            // Formatted table
console.trace();                          // Stack trace
console.time('label'); /* code */ console.timeEnd('label');  // Timing
console.group('Section'); /* logs */ console.groupEnd();     // Grouping

Node.js Debugging

# Start with inspector
node --inspect dist/index.js

# Break on first line
node --inspect-brk dist/index.js

# Then open Chrome DevTools at chrome://inspect

React DevTools

• Components tab: Inspect props/state
• Profiler tab: Find re-render issues

Git Investigation Commands

# Who changed this line?
git blame path/to/file

# What changed in last 5 commits for this file?
git log --oneline -5 -- path/to/file

# Full diff of what changed
git diff HEAD~5 -- path/to/file

# Find when a specific string was added/removed
git log -S "searchString" --oneline

# Find commits by a specific author
git log --author="name" --oneline -10

Performance Debugging

Identify Slow Code

console.time('operation');
// ... code to measure
console.timeEnd('operation');

Memory Leaks

// Common causes:
// 1. Uncleared intervals/timeouts
// 2. Event listeners not removed
// 3. Closures holding references
// 4. Growing arrays/maps

// Fix: Always cleanup
useEffect(() => {
  const handler = () => {};
  window.addEventListener('resize', handler);
  return () => window.removeEventListener('resize', handler);  // Cleanup!
}, []);

Database Query Performance

-- Check execution plan
SET SHOWPLAN_TEXT ON;
GO
SELECT ... FROM ...;
GO
SET SHOWPLAN_TEXT OFF;

-- Look for: Table scans (need index), high estimated rows, sort operations
-- Fix: Add indexes, limit result sets, use specific columns instead of SELECT *

Bug Report Format

When documenting bugs:

## Bug: [Short description]

### Steps to Reproduce
1. [Step 1]
2. [Step 2]
3. [Step 3]

### Expected Behavior
[What should happen]

### Actual Behavior
[What actually happens]

### Environment
- System: [D365/Maximo/Fabric/Local/VPS]
- OS: [macOS/Windows/Linux]
- Node: [version]
- Browser: [if applicable]
- Database: [connection used]
- Branch: [git branch]

### Evidence Gathered
[Logs, query results, diffs -- concrete data, not assumptions]

### Root Cause
[After investigation, with evidence reference -- NOT speculation]

### Fix
[Solution applied -- minimal and targeted]

### Verification
[Proof that fix works -- test output, query results, screenshots]

Verification Checklist

• STOP-THINK-ACT executed (Phase 0 complete)
• System/environment confirmed with user
• Output format matches user's request
• Database schema verified before writing queries (if applicable)
• Bug reproduced with specific steps
• Evidence gathered before any conclusions stated
• Root cause identified with supporting evidence (not speculation)
• No unproven attribution of blame
• Fix is minimal and targeted (no unrelated changes)
• Fix verified to work (test output shown)
• No regressions introduced (existing tests pass)
• Secondary config files checked
• Investigation stayed within budget (not spiraling)

Quick Reference: Friction Prevention

Key Principle

STOP before acting. Confirm context. Gather evidence. Show findings before conclusions. Fix what's proven broken.

Don't guess. Don't assume which system. Don't attribute causes without diffs. Don't explore when asked to fix. Don't deliver the wrong format. The bug tells you what's wrong if you listen -- but only after you've confirmed you're looking at the right system, the right file, and the right data.