mchittineni SRE Incident Command & Blameless Post-Mortem Standard
When to Use This Skill
Triggers — load this skill when:
- An active outage needs roles, comms cadence, and mitigation discipline
- A severity matrix or escalation policy must be defined or applied
- A blameless post-mortem and its action items need to be written and tracked
Route elsewhere when:
- Security compromise rather than availability failure ->
secops-incident-triage-forensics - Error-budget policy that triggers freezes ->
sli-slo-error-budget-design - DR failover execution ->
backup-and-disaster-recovery
1. Incident Command System (ICS) Roles
[Incident Commander (IC)]
|
+------------------+------------------+
| |
[Operations Lead] [Communications Lead]
Executes diagnostic checks Updates statuspage, executives,
and deployment rollbacks and customer support channels
2. Severity Classification Matrix
| Severity | Customer Impact | MTTD Target | MTTR Target | Action Required |
|---|---|---|---|---|
| SEV-1 | Critical system down (>25% users affected / financial loss) | < 5 mins | < 30 mins | Immediate all-hands bridge, executive notification every 30m |
| SEV-2 | Significant degradation / core feature failing | < 15 mins | < 2 hours | Dedicated Slack channel + SRE bridge |
| SEV-3 | Minor non-blocking issue or single customer affected | < 1 hour | < 1 business day | Standard ticket queue triage |
3. Blameless Post-Mortem Template
# Incident Post-Mortem: [YYYY-MM-DD] [Incident Title]
## Executive Summary
- **Impact Duration**: 42 minutes (14:12 UTC to 14:54 UTC)
- **User Impact**: ~14,200 checkout attempts failed with HTTP 504.
- **Root Cause**: Database connection pool exhaustion caused by missing query index during peak traffic burst.
## Timeline (UTC)
- **14:12** - Latency alert fires for `checkout-service`.
- **14:15** - Incident declared SEV-1; Incident Commander appointed.
- **14:28** - Operations lead discovers 100% DB thread pool saturation.
- **14:38** - Hotfix index deployed to RDS reader/writer.
- **14:54** - Latency and error rates return to nominal SLO levels.
## Lessons Learned & Action Items
| Action Item | Type | Owner | Priority | Target Date |
|---|---|---|---|---|
| Implement query timeout limit on checkout repo | Preventative | @alice | P1 | Next Sprint |
| Add DB connection pool saturation SLO alert | Detection | @bob | P1 | 3 Days |
4. Scribe Discipline & Role Handoff
The Scribe is not optional. Without one, the timeline is reconstructed afterwards from memory and Slack scroll-back, which is exactly when detail is lost and blame appears.
The Scribe records, in the incident channel, with timestamps in UTC:
- every state change ("error rate back under 1%"), not commentary;
- every action taken and by whom, including the ones that did not work;
- every hypothesis raised and how it was ruled in or out;
- every decision the Incident Commander makes, and the time it was made.
That log is the post-mortem's timeline section, written for free during the incident.
Explicit handoff. Fatigue causes second incidents. Every role hands over out loud and in writing after ~2 hours or at a shift boundary:
HANDOFF 14:05Z — Incident Commander: @alex -> @sam
Current state: SEV2, error rate 4% (was 18%), degraded checkout only.
Working theory: connection-pool exhaustion after the 12:40Z deploy.
In flight: @dana rolling back to build 4471 (ETA 14:15Z).
Ruled out: DNS, upstream provider, node pressure.
Next decision point: 14:20Z — if rollback does not clear it, escalate to SEV1 and page DBRE.
The outgoing IC states it, the incoming IC confirms in channel, and the Scribe logs both. No silent handoffs, and never more than one Incident Commander at a time.