Root Cause Analysis Skill
If you fixed it but it came back, you fixed a symptom.
Core Principle
Every symptom has a cause. Every cause has a deeper cause. Keep digging until you reach something you can prevent, not just fix.
5 Whys — Extended Example
| # |
Question |
Answer |
| 1 |
Why did the page crash? |
JavaScript threw a TypeError on null |
| 2 |
Why was the value null? |
The API returned an empty response |
| 3 |
Why did the API return empty? |
The database query timed out |
| 4 |
Why did the query time out? |
Missing index on a 10M-row table |
| 5 |
Why was the index missing? |
No performance review in the PR process |
Root cause: Process gap (no performance review), not the missing index.
Fix the system: Add performance checklist to PR template, not just add the index.
5 Whys Traps
| Trap |
Example |
How to Avoid |
| Stopping at human error |
"Dev forgot to add the index" |
Ask why was it possible to forget? |
| Single chain only |
Only follow one branch |
Branch at each Why if multiple causes |
| Speculation without evidence |
"Probably because of..." |
Each answer must have evidence |
| Going too deep |
Why #12: "Because physics" |
Stop when you reach an actionable system change |
Cause Categories
| Category |
Common Patterns |
Investigation Tools |
| Code |
Null reference, off-by-one, race condition, type mismatch |
Debugger, unit tests, static analysis |
| Data |
Corrupt input, unexpected format, encoding issues |
Query logs, data validation, sample inspection |
| Infrastructure |
Disk full, memory exhaustion, network partition |
Metrics dashboards, health endpoints, top/df |
| Dependencies |
Breaking change, version mismatch, transitive conflict |
Lockfile diff, changelog review, npm ls |
| Configuration |
Wrong env var, feature flag state, missing secret |
Config diff, environment comparison |
| Process |
Missing review, unclear ownership, no runbook |
Post-mortem patterns, team interviews |
Investigation Techniques
Binary Search Debugging
When you don't know where the bug is, halve the search space:
- Identify the last known good state (commit, deploy, timestamp)
git bisect between good and bad- Each step: does the bug exist? Yes → go earlier. No → go later.
- Result: the exact commit that introduced the bug.
Timeline Reconstruction
| Time |
Event |
Source |
| T-24h |
Deploy v2.3.1 |
CI/CD logs |
| T-12h |
Config change: cache TTL 60→30s |
Config audit log |
| T-2h |
First user report |
Support tickets |
| T-0 |
Alert fired |
Monitoring |
Key question: What changed between "working" and "broken"?
Correlation vs Causation
| Evidence Type |
Confidence |
Example |
| Reproduces on demand |
High |
"Every time I submit this form..." |
| Correlates with a deploy |
Medium |
"Started after we deployed" |
| Timing coincidence |
Low |
"Started Monday" (traffic patterns?) |
| "It's never done this before" |
Very Low |
Memory is unreliable — check logs |
Fix + Prevent Pattern
| Phase |
Purpose |
Example |
Deadline |
| Immediate |
Stop the bleeding |
Rollback, disable feature, redirect traffic |
Now |
| Permanent |
Fix root cause |
Add missing index, fix validation, patch dependency |
This sprint |
| Prevention |
Stop recurrence |
Add CI check, monitoring alert, runbook, PR checklist |
Next sprint |
Test the fix: The permanent fix should make the immediate fix unnecessary. If you remove the band-aid and the symptom returns, you haven't found root cause.
Common Symptom → Root Cause Patterns
| Symptom |
Obvious Cause |
Deeper Root Cause |
| Memory leak |
Unclosed resource |
No resource cleanup pattern in codebase |
| N+1 queries |
Missing join |
ORM hides query count, no query logging |
| Intermittent test failure |
Timing-dependent |
Shared mutable state between tests |
| "Works on my machine" |
Different environment |
No environment parity tooling (Docker, etc.) |
| Data corruption |
Missing validation |
Validation in UI only, not at API boundary |
| Slow deploys |
Large artifact |
No build caching, monorepo without selective builds |
Post-Mortem Integration
The RCA section of a post-mortem should include:
- The 5 Whys chain (with evidence for each level)
- Contributing factors (things that made it worse, not the direct cause)
- What we were lucky about (things that could have made it much worse)
- Action items with owners and dates for permanent fix + prevention
Synapses
See synapses.json for connections.
1---2name: root-cause-analysis3description: Find the true source, not symptoms — systematic debugging from observation to permanent fix4---56# Root Cause Analysis Skill78> If you fixed it but it came back, you fixed a symptom.910## Core Principle1112Every symptom has a cause. Every cause has a deeper cause. Keep digging until you reach something you can *prevent*, not just fix.1314## 5 Whys — Extended Example1516| # | Question | Answer |17| - | -------- | ------ |18| 1 | Why did the page crash? | JavaScript threw a TypeError on null |19| 2 | Why was the value null? | The API returned an empty response |20| 3 | Why did the API return empty? | The database query timed out |21| 4 | Why did the query time out? | Missing index on a 10M-row table |22| 5 | Why was the index missing? | No performance review in the PR process |2324**Root cause**: Process gap (no performance review), not the missing index.25**Fix the system**: Add performance checklist to PR template, not just add the index.2627### 5 Whys Traps2829| Trap | Example | How to Avoid |30| ---- | ------- | ------------ |31| Stopping at human error | "Dev forgot to add the index" | Ask *why was it possible to forget?* |32| Single chain only | Only follow one branch | Branch at each Why if multiple causes |33| Speculation without evidence | "Probably because of..." | Each answer must have evidence |34| Going too deep | Why #12: "Because physics" | Stop when you reach an actionable system change |3536## Cause Categories3738| Category | Common Patterns | Investigation Tools |39| -------- | --------------- | ------------------- |40| Code | Null reference, off-by-one, race condition, type mismatch | Debugger, unit tests, static analysis |41| Data | Corrupt input, unexpected format, encoding issues | Query logs, data validation, sample inspection |42| Infrastructure | Disk full, memory exhaustion, network partition | Metrics dashboards, health endpoints, `top`/`df` |43| Dependencies | Breaking change, version mismatch, transitive conflict | Lockfile diff, changelog review, `npm ls` |44| Configuration | Wrong env var, feature flag state, missing secret | Config diff, environment comparison |45| Process | Missing review, unclear ownership, no runbook | Post-mortem patterns, team interviews |4647## Investigation Techniques4849### Binary Search Debugging5051When you don't know where the bug is, halve the search space:52531. Identify the last known good state (commit, deploy, timestamp)542. `git bisect` between good and bad553. Each step: does the bug exist? Yes → go earlier. No → go later.564. Result: the exact commit that introduced the bug.5758### Timeline Reconstruction5960| Time | Event | Source |61| ---- | ----- | ------ |62| T-24h | Deploy v2.3.1 | CI/CD logs |63| T-12h | Config change: cache TTL 60→30s | Config audit log |64| T-2h | First user report | Support tickets |65| T-0 | Alert fired | Monitoring |6667**Key question**: What changed between "working" and "broken"?6869### Correlation vs Causation7071| Evidence Type | Confidence | Example |72| ------------- | ---------- | ------- |73| Reproduces on demand | High | "Every time I submit this form..." |74| Correlates with a deploy | Medium | "Started after we deployed" |75| Timing coincidence | Low | "Started Monday" (traffic patterns?) |76| "It's never done this before" | Very Low | Memory is unreliable — check logs |7778## Fix + Prevent Pattern7980| Phase | Purpose | Example | Deadline |81| ----- | ------- | ------- | -------- |82| **Immediate** | Stop the bleeding | Rollback, disable feature, redirect traffic | Now |83| **Permanent** | Fix root cause | Add missing index, fix validation, patch dependency | This sprint |84| **Prevention** | Stop recurrence | Add CI check, monitoring alert, runbook, PR checklist | Next sprint |8586**Test the fix**: The permanent fix should make the immediate fix unnecessary. If you remove the band-aid and the symptom returns, you haven't found root cause.8788## Common Symptom → Root Cause Patterns8990| Symptom | Obvious Cause | Deeper Root Cause |91| ------- | ------------- | ----------------- |92| Memory leak | Unclosed resource | No resource cleanup pattern in codebase |93| N+1 queries | Missing join | ORM hides query count, no query logging |94| Intermittent test failure | Timing-dependent | Shared mutable state between tests |95| "Works on my machine" | Different environment | No environment parity tooling (Docker, etc.) |96| Data corruption | Missing validation | Validation in UI only, not at API boundary |97| Slow deploys | Large artifact | No build caching, monorepo without selective builds |9899## Post-Mortem Integration100101The RCA section of a post-mortem should include:1021031. **The 5 Whys chain** (with evidence for each level)1042. **Contributing factors** (things that made it worse, not the direct cause)1053. **What we were lucky about** (things that could have made it much worse)1064. **Action items** with owners and dates for permanent fix + prevention107108## Synapses109110See [synapses.json](synapses.json) for connections.
diegosouzapw