revfactory RCA Methodology — Root Cause Analysis Methodology Guide
A collection of structured analysis techniques for systematically tracing incident root causes.
1. 5 Whys Technique
Procedure
Problem: Payment service was down for 30 minutes.
Why 1: Why did the payment service go down?
-> The DB connection pool was exhausted.
Why 2: Why was the connection pool exhausted?
-> Slow queries held connections for extended periods.
Why 3: Why did slow queries occur?
-> A full table scan was performed without an index.
Why 4: Why was there no index?
-> The index addition was missed in the migration during the new feature deployment.
Why 5: Why was the missing index not detected?
-> Query performance verification was not included in the deployment pipeline.
Root Cause: Absence of query performance verification step in the deployment pipeline
5 Whys Pitfalls
| Pitfall | Description | Prevention |
|---|---|---|
| Stopping too early | Concluding at step 2-3 | Verify: "Would fixing this prevent recurrence?" |
| Leading to blame | Ending with "who made the mistake" | Focus on system/process causes |
| Single path only | Missing compound causes | Review branches at each step |
| Speculation-based answers | Hypotheses without evidence | Verify with logs/metrics |
2. Fishbone Diagram (Ishikawa)
+- People -------- Insufficient deployer training
| Code review skipped
|
+- Process -------- Deployment checklist not followed
| Performance testing not conducted
|
Payment Svc Down <--+- Technology ---- Missing index
| Connection pool size insufficient
|
+- Environment ---- Production data scale not reflected
| Staging environment mismatch
|
+- Monitoring ----- Slow query alert not configured
DB metrics dashboard absent
6M Categories (Software Application)
| Traditional 6M | Software Application | Investigation Items |
|---|---|---|
| Man | People/Team | Training, communication, on-call structure |
| Method | Process | Deployment procedures, change management, approval flows |
| Machine | Technology/Infrastructure | Servers, DB, network, code |
| Material | Data/Input | Input data, external API responses |
| Measurement | Monitoring | Alerts, metrics, logs, tracing |
| Environment | Environment | Configuration, env vars, dependent services |
3. Fault Tree Analysis (FTA)
Payment Service Outage (Top Event)
|
+-----| OR |-----+
| |
DB Failure Application Failure
| |
+-----| OR |-----+ +---| AND |---+
| | | |
Pool Exhaustion Disk Full Slow Queries High Traffic
| |
+-----| AND |-----+ +-----| OR |-----+
| | | |
Missing Index Large Data New Code Deploy Schema Change
Probability Calculation:
OR gate: P(A OR B) = 1 - (1-P(A)) x (1-P(B))
AND gate: P(A AND B) = P(A) x P(B)
Example: P(slow query)=0.3, P(high traffic)=0.2
P(app failure) = 0.3 x 0.2 = 0.06 (6%)
4. Change Analysis
Investigate changes before and after the incident:
| Change Time | Change Content | Changed By | Impact Scope | Incident Correlation |
|------------|---------------|-----------|-------------|---------------------|
| T-2h | Payment API v2.3 deploy | Dev team | Payment service | HIGH |
| T-1h | Redis config change | Ops team | Cache service | MEDIUM |
| T-30m | CDN cache purge | Marketing | Frontend | LOW |
Correlation criteria:
1. Temporal proximity: Change time <-> Incident time
2. Scope match: Change scope <-> Incident impact scope
3. Rollback effect: Whether rolling back the change resolves the incident
Cognitive Bias Prevention Checklist
| Bias | Description | Prevention |
|---|---|---|
| Confirmation bias | Collecting only evidence matching the first hypothesis | Actively search for counterexamples |
| Hindsight bias | "Obviously this was the cause" | Judge based only on information available at the time |
| Availability bias | Equating with recently experienced similar incidents | Enforce evidence-based analysis |
| Fundamental attribution error | Attributing to human error | Prioritize system/process causes |
| Anchoring | Fixating on initial report content | Analyze independently from multiple perspectives |
RCA Technique Selection Guide
| Situation | Recommended Technique | Reason |
|---|---|---|
| Simple incident, quick analysis needed | 5 Whys | Lightweight, can be done immediately |
| Suspected compound causes | Fishbone | Multi-dimensional cause exploration |
| Safety-related severe incidents | FTA | Quantitative, systematic |
| Post-deployment incidents | Change Analysis | Narrow down cause candidates |
| All cases | 5 Whys + Change Analysis combined | Fast yet systematic |