Root Cause Analysis

Root Cause Analysis Skill

If you fixed it but it came back, you fixed a symptom.

Core Principle

Every symptom has a cause. Every cause has a deeper cause. Keep digging until you reach something you can prevent, not just fix.

5 Whys — Extended Example

Root cause: Process gap (no performance review), not the missing index. Fix the system: Add performance checklist to PR template, not just add the index.

5 Whys Traps

Cause Categories

Investigation Techniques

Binary Search Debugging

When you don't know where the bug is, halve the search space:

1. Identify the last known good state (commit, deploy, timestamp)
2. git bisect between good and bad
3. Each step: does the bug exist? Yes → go earlier. No → go later.
4. Result: the exact commit that introduced the bug.

# Binary search with git bisect
git bisect start
git bisect bad HEAD          # Current commit is broken
git bisect good v2.3.0       # This tag was working

# Git checks out middle commit, you test
# Repeat: git bisect good/bad until found
git bisect reset             # Return to HEAD when done

// Binary search debugging in code
async function findBreakingChange(
  commits: string[],
  testFn: (commit: string) => Promise<boolean>
): Promise<string | null> {
  let left = 0;
  let right = commits.length - 1;
  
  while (left < right) {
    const mid = Math.floor((left + right) / 2);
    const works = await testFn(commits[mid]);
    
    if (works) {
      left = mid + 1;  // Bug introduced after this commit
    } else {
      right = mid;     // Bug exists at or before this commit
    }
  }
  
  return commits[left] ?? null;
}

Timeline Reconstruction

Key question: What changed between "working" and "broken"?

Correlation vs Causation

Fix + Prevent Pattern

Test the fix: The permanent fix should make the immediate fix unnecessary. If you remove the band-aid and the symptom returns, you haven't found root cause.

Common Symptom → Root Cause Patterns

Post-Mortem Integration

The RCA section of a post-mortem should include:

1. The 5 Whys chain (with evidence for each level)
2. Contributing factors (things that made it worse, not the direct cause)
3. What we were lucky about (things that could have made it much worse)
4. Action items with owners and dates for permanent fix + prevention