Root Cause Analysis Skill
If you fixed it but it came back, you fixed a symptom.
Core Principle
Every symptom has a cause. Every cause has a deeper cause. Keep digging until you reach something you can prevent, not just fix.
5 Whys — Extended Example
| # |
Question |
Answer |
| 1 |
Why did the page crash? |
JavaScript threw a TypeError on null |
| 2 |
Why was the value null? |
The API returned an empty response |
| 3 |
Why did the API return empty? |
The database query timed out |
| 4 |
Why did the query time out? |
Missing index on a 10M-row table |
| 5 |
Why was the index missing? |
No performance review in the PR process |
Root cause: Process gap (no performance review), not the missing index.
Fix the system: Add performance checklist to PR template, not just add the index.
5 Whys Traps
| Trap |
Example |
How to Avoid |
| Stopping at human error |
"Dev forgot to add the index" |
Ask why was it possible to forget? |
| Single chain only |
Only follow one branch |
Branch at each Why if multiple causes |
| Speculation without evidence |
"Probably because of..." |
Each answer must have evidence |
| Going too deep |
Why #12: "Because physics" |
Stop when you reach an actionable system change |
Cause Categories
| Category |
Common Patterns |
Investigation Tools |
| Code |
Null reference, off-by-one, race condition, type mismatch |
Debugger, unit tests, static analysis |
| Data |
Corrupt input, unexpected format, encoding issues |
Query logs, data validation, sample inspection |
| Infrastructure |
Disk full, memory exhaustion, network partition |
Metrics dashboards, health endpoints, top/df |
| Dependencies |
Breaking change, version mismatch, transitive conflict |
Lockfile diff, changelog review, npm ls |
| Configuration |
Wrong env var, feature flag state, missing secret |
Config diff, environment comparison |
| Process |
Missing review, unclear ownership, no runbook |
Post-mortem patterns, team interviews |
Investigation Techniques
Binary Search Debugging
When you don't know where the bug is, halve the search space:
- Identify the last known good state (commit, deploy, timestamp)
git bisect between good and bad
- Each step: does the bug exist? Yes → go earlier. No → go later.
- Result: the exact commit that introduced the bug.
# Binary search with git bisect
git bisect start
git bisect bad HEAD # Current commit is broken
git bisect good v2.3.0 # This tag was working
# Git checks out middle commit, you test
# Repeat: git bisect good/bad until found
git bisect reset # Return to HEAD when done
// Binary search debugging in code
async function findBreakingChange(
commits: string[],
testFn: (commit: string) => Promise<boolean>
): Promise<string | null> {
let left = 0;
let right = commits.length - 1;
while (left < right) {
const mid = Math.floor((left + right) / 2);
const works = await testFn(commits[mid]);
if (works) {
left = mid + 1; // Bug introduced after this commit
} else {
right = mid; // Bug exists at or before this commit
}
}
return commits[left] ?? null;
}
Timeline Reconstruction
| Time |
Event |
Source |
| T-24h |
Deploy v2.3.1 |
CI/CD logs |
| T-12h |
Config change: cache TTL 60→30s |
Config audit log |
| T-2h |
First user report |
Support tickets |
| T-0 |
Alert fired |
Monitoring |
Key question: What changed between "working" and "broken"?
Correlation vs Causation
| Evidence Type |
Confidence |
Example |
| Reproduces on demand |
High |
"Every time I submit this form..." |
| Correlates with a deploy |
Medium |
"Started after we deployed" |
| Timing coincidence |
Low |
"Started Monday" (traffic patterns?) |
| "It's never done this before" |
Very Low |
Memory is unreliable — check logs |
Fix + Prevent Pattern
| Phase |
Purpose |
Example |
Deadline |
| Immediate |
Stop the bleeding |
Rollback, disable feature, redirect traffic |
Now |
| Permanent |
Fix root cause |
Add missing index, fix validation, patch dependency |
This sprint |
| Prevention |
Stop recurrence |
Add CI check, monitoring alert, runbook, PR checklist |
Next sprint |
Test the fix: The permanent fix should make the immediate fix unnecessary. If you remove the band-aid and the symptom returns, you haven't found root cause.
Common Symptom → Root Cause Patterns
| Symptom |
Obvious Cause |
Deeper Root Cause |
| Memory leak |
Unclosed resource |
No resource cleanup pattern in codebase |
| N+1 queries |
Missing join |
ORM hides query count, no query logging |
| Intermittent test failure |
Timing-dependent |
Shared mutable state between tests |
| "Works on my machine" |
Different environment |
No environment parity tooling (Docker, etc.) |
| Data corruption |
Missing validation |
Validation in UI only, not at API boundary |
| Slow deploys |
Large artifact |
No build caching, monorepo without selective builds |
Post-Mortem Integration
The RCA section of a post-mortem should include:
- The 5 Whys chain (with evidence for each level)
- Contributing factors (things that made it worse, not the direct cause)
- What we were lucky about (things that could have made it much worse)
- Action items with owners and dates for permanent fix + prevention
1---2name: root-cause-analysis-43description: "Find the true source, not symptoms — systematic debugging from observation to permanent fix"4tier: core5applyTo: "**/*debug*,**/*error*,**/*bug*,**/*issue*,**/*problem*"6currency: 2026-04-207metadata:8 source: https://skillmd.com/skills/fabioc-aloha/root-cause-analysis-49---1011# Root Cause Analysis Skill1213> If you fixed it but it came back, you fixed a symptom.1415## Core Principle1617Every symptom has a cause. Every cause has a deeper cause. Keep digging until you reach something you can *prevent*, not just fix.1819## 5 Whys — Extended Example2021| # | Question | Answer |22| - | -------- | ------ |23| 1 | Why did the page crash? | JavaScript threw a TypeError on null |24| 2 | Why was the value null? | The API returned an empty response |25| 3 | Why did the API return empty? | The database query timed out |26| 4 | Why did the query time out? | Missing index on a 10M-row table |27| 5 | Why was the index missing? | No performance review in the PR process |2829**Root cause**: Process gap (no performance review), not the missing index.30**Fix the system**: Add performance checklist to PR template, not just add the index.3132### 5 Whys Traps3334| Trap | Example | How to Avoid |35| ---- | ------- | ------------ |36| Stopping at human error | "Dev forgot to add the index" | Ask *why was it possible to forget?* |37| Single chain only | Only follow one branch | Branch at each Why if multiple causes |38| Speculation without evidence | "Probably because of..." | Each answer must have evidence |39| Going too deep | Why #12: "Because physics" | Stop when you reach an actionable system change |4041## Cause Categories4243| Category | Common Patterns | Investigation Tools |44| -------- | --------------- | ------------------- |45| Code | Null reference, off-by-one, race condition, type mismatch | Debugger, unit tests, static analysis |46| Data | Corrupt input, unexpected format, encoding issues | Query logs, data validation, sample inspection |47| Infrastructure | Disk full, memory exhaustion, network partition | Metrics dashboards, health endpoints, `top`/`df` |48| Dependencies | Breaking change, version mismatch, transitive conflict | Lockfile diff, changelog review, `npm ls` |49| Configuration | Wrong env var, feature flag state, missing secret | Config diff, environment comparison |50| Process | Missing review, unclear ownership, no runbook | Post-mortem patterns, team interviews |5152## Investigation Techniques5354### Binary Search Debugging5556When you don't know where the bug is, halve the search space:57581. Identify the last known good state (commit, deploy, timestamp)592. `git bisect` between good and bad603. Each step: does the bug exist? Yes → go earlier. No → go later.614. Result: the exact commit that introduced the bug.6263```bash64# Binary search with git bisect65git bisect start66git bisect bad HEAD # Current commit is broken67git bisect good v2.3.0 # This tag was working6869# Git checks out middle commit, you test70# Repeat: git bisect good/bad until found71git bisect reset # Return to HEAD when done72```7374```typescript75// Binary search debugging in code76async function findBreakingChange(77 commits: string[],78 testFn: (commit: string) => Promise<boolean>79): Promise<string | null> {80 let left = 0;81 let right = commits.length - 1;82 83 while (left < right) {84 const mid = Math.floor((left + right) / 2);85 const works = await testFn(commits[mid]);86 87 if (works) {88 left = mid + 1; // Bug introduced after this commit89 } else {90 right = mid; // Bug exists at or before this commit91 }92 }93 94 return commits[left] ?? null;95}96```9798### Timeline Reconstruction99100| Time | Event | Source |101| ---- | ----- | ------ |102| T-24h | Deploy v2.3.1 | CI/CD logs |103| T-12h | Config change: cache TTL 60→30s | Config audit log |104| T-2h | First user report | Support tickets |105| T-0 | Alert fired | Monitoring |106107**Key question**: What changed between "working" and "broken"?108109### Correlation vs Causation110111| Evidence Type | Confidence | Example |112| ------------- | ---------- | ------- |113| Reproduces on demand | High | "Every time I submit this form..." |114| Correlates with a deploy | Medium | "Started after we deployed" |115| Timing coincidence | Low | "Started Monday" (traffic patterns?) |116| "It's never done this before" | Very Low | Memory is unreliable — check logs |117118## Fix + Prevent Pattern119120| Phase | Purpose | Example | Deadline |121| ----- | ------- | ------- | -------- |122| **Immediate** | Stop the bleeding | Rollback, disable feature, redirect traffic | Now |123| **Permanent** | Fix root cause | Add missing index, fix validation, patch dependency | This sprint |124| **Prevention** | Stop recurrence | Add CI check, monitoring alert, runbook, PR checklist | Next sprint |125126**Test the fix**: The permanent fix should make the immediate fix unnecessary. If you remove the band-aid and the symptom returns, you haven't found root cause.127128## Common Symptom → Root Cause Patterns129130| Symptom | Obvious Cause | Deeper Root Cause |131| ------- | ------------- | ----------------- |132| Memory leak | Unclosed resource | No resource cleanup pattern in codebase |133| N+1 queries | Missing join | ORM hides query count, no query logging |134| Intermittent test failure | Timing-dependent | Shared mutable state between tests |135| "Works on my machine" | Different environment | No environment parity tooling (Docker, etc.) |136| Data corruption | Missing validation | Validation in UI only, not at API boundary |137| Slow deploys | Large artifact | No build caching, monorepo without selective builds |138139## Post-Mortem Integration140141The RCA section of a post-mortem should include:1421431. **The 5 Whys chain** (with evidence for each level)1442. **Contributing factors** (things that made it worse, not the direct cause)1453. **What we were lucky about** (things that could have made it much worse)1464. **Action items** with owners and dates for permanent fix + prevention