saitarrun Skill: Incident Management
Systematic incident response and postmortem process for production issues. Emphasizes speed (MTTR), clarity (timeline), and learning (blameless culture).
Incident Classification (SEV Levels)
SEV 1 (Critical)
- Entire service down OR
- Core functionality unavailable AND
- Large user base affected AND
- Revenue/reputation impact
- MTTR target: < 30 minutes
- Declare: Immediately
- Escalation: Engineering Manager + Tech Lead + On-Call lead
SEV 2 (High)
- Partial degradation (slow, some users affected) OR
- Non-core features completely down
- MTTR target: < 2 hours
- Declare: Within 15 min
- Escalation: Tech Lead + Service Owner
SEV 3 (Medium)
- Non-critical feature broken OR
- Workaround exists AND
- Low user impact
- MTTR target: < 24 hours
- Declare: Next business day
- Escalation: Service Owner
SEV 4 (Low)
- Minor issue, cosmetic OR
- Zero user impact
- Can wait for next release
- Escalation: None (treat as bug)
Incident Timeline Template
## Incident: [Title]
### Summary
- Service: [service-name]
- Duration: [start] - [end] (X minutes)
- Users affected: [X]%
- Severity: SEV [1-4]
### Timeline
| Time | Event | Owner |
|------|-------|-------|
| 14:32 UTC | ❌ Alert: API error rate > 10% | Monitoring |
| 14:33 UTC | 🟡 Incident declared (SEV 2) | On-Call |
| 14:35 UTC | 🔍 Root cause identified: DB connection pool exhausted | Backend lead |
| 14:37 UTC | 🔧 Mitigation: Increased pool size, restarted service | DevOps |
| 14:39 UTC | ✅ Incident resolved, metrics normal | Monitoring |
| 14:45 UTC | ✅ All systems nominal (monitoring for 10 min) | On-Call |
### Root Cause
Database connection pool was set to 50 but app scaled to 8 instances (50 x 8 = 400 connections but pool limit was 50). New deployment didn't adjust pool size.
### Contributing Factors
1. No automated pool size calculation (manual config)
2. Pre-deployment validation didn't catch pool exhaustion risk
3. Alerting was on error rate, not pool usage (too late)
### Resolution
1. Temporarily increased pool size from 50 to 200 (immediate)
2. Disabled auto-scaling during hot-fix (reduced concurrent instances)
### Monitoring During Incident
- API error rate: 8% → 0.2% after fix
- DB response time: 2s → 50ms after fix
- User-visible errors: Resolved
### Follow-up Actions
1. [Action] Implement auto-sizing of connection pool based on instance count
2. [Action] Add alert on pool usage % (critical at 80%+)
3. [Action] Add pre-deployment validation check for pool size
4. [Action] Document connection pool sizing in runbook
Postmortem Process (Blameless)
Before Postmortem
- Wait 24-48 hours (emotions settle, facts clear)
- Send survey to participants (get thoughts while fresh)
- Collect logs, metrics, alerts
During Postmortem (60 min)
## Postmortem: [Service] incident on [Date]
### 1. Incident Summary (5 min)
[What happened, impact, duration]
### 2. Timeline (10 min)
[When did each major event occur? Focus on facts, not blame]
### 3. Root Cause Analysis (15 min)
**Ask "Why?" 5 times**:
1. Why did error rate spike?
→ Connection pool exhausted
2. Why was pool exhausted?
→ Pool size fixed at 50, scaled to 8 instances
3. Why wasn't pool size updated?
→ No automated calculation, manual config
4. Why wasn't this caught in testing?
→ Load tests didn't use production instance count
5. Why don't we catch this automatically?
→ Pre-deployment validation doesn't check pool usage
**Root cause**: No automated pool sizing based on instance count
### 4. Contributing Factors (not root causes)
- Alert on error rate was too late (pool full = errors immediate)
- No pre-deployment pool validation
- Runbook didn't document pool sizing
### 5. Action Items (10 min)
| Action | Owner | Deadline | Priority |
|--------|-------|----------|----------|
| Auto-size pool based on instances | Backend lead | 2 weeks | P0 |
| Add pool usage alert (80%+ critical) | Oncall | 1 week | P0 |
| Add pre-deploy validation check | DevOps | 1 week | P1 |
| Document pool sizing in runbook | Tech writer | 3 days | P1 |
### 6. Prevention Ideas (10 min)
[Brainstorm how to prevent similar incidents]
### 7. Blameless Check (5 min)
- Did we focus on facts/systems, not people? ✓
- Did we assume good intent? ✓
- Are all participants comfortable with the report? ✓
After Postmortem
- Distribute report to all team (transparency)
- Track action items (check in 2 weeks)
- Close the loop (what did we learn?)
RCA (Root Cause Analysis) Framework
Don't stop at the symptom:
❌ Root cause: Operator error
✓ Root cause: No automated validation → human forgot step
❌ Root cause: Connection pool too small
✓ Root cause: Pool size is manual config, no auto-sizing based on scale
❌ Root cause: Developer didn't check metrics
✓ Root cause: No alert on pool usage %, alert was on error rate too late
Blameless Culture Principles
- Assume good intent — Engineer did their best with information available
- Focus on systems — What failed in the process/tooling/documentation?
- No "stupid" mistakes — If someone made a mistake, how do we prevent it systematically?
- Psychological safety — Team must feel safe to admit mistakes and learn
- Shared responsibility — Incident is team's problem, not individual's
- Action items — Every postmortem must end with concrete improvements
Incident Communication
During incident (update every 5-10 minutes):
🟡 INCIDENT: API service degraded
Status: INVESTIGATING
Impact: Users may experience 50% error rate
Last update: 14:35 UTC
Next update: 14:40 UTC
https://status.example.com/incidents/[id]
When resolved:
✅ RESOLVED: API service
Status: RESOLVED (14:39 UTC)
Duration: 7 minutes
Impact: ~2000 users affected
Postmortem link: [link]
Thank you for your patience.
Metrics to Track
- MTTR (Mean Time To Recovery): Target < 30 min SEV1
- Detection time: How fast did we notice?
- Resolution time: How fast did we fix?
- Repeat incidents: Same root cause within 6 months?
- Postmortem quality: All action items tracked?
- Prevention success: Did fixes prevent recurrence?
Status: Ready for production incident response
Best for: Production incidents, postmortems, RCA, blameless culture