# Incident Management And Postmortem

> SRE incident command: ICS role assignment (IC, Ops, Comms, Scribe), severity matrix, live mitigation and comms protocol, and blameless post-mortem with tracked action items. Use when running or improving incident response, declaring severity, coordinating an active outage, or writing a post-mortem.

- Skill: `mchittineni/incident-management-and-postmortem` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add mchittineni/incident-management-and-postmortem`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mchittineni/incident-management-and-postmortem/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: mchittineni (https://skillmd.com/u/mchittineni)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/mchittineni/incident-management-and-postmortem

---


# SRE Incident Command & Blameless Post-Mortem Standard

## When to Use This Skill

**Triggers — load this skill when:**

- An active outage needs roles, comms cadence, and mitigation discipline
- A severity matrix or escalation policy must be defined or applied
- A blameless post-mortem and its action items need to be written and tracked

**Route elsewhere when:**

- Security compromise rather than availability failure -> `secops-incident-triage-forensics`
- Error-budget policy that triggers freezes -> `sli-slo-error-budget-design`
- DR failover execution -> `backup-and-disaster-recovery`

## 1. Incident Command System (ICS) Roles

```text
                   [Incident Commander (IC)]
                               |
            +------------------+------------------+
            |                                     |
    [Operations Lead]                      [Communications Lead]
  Executes diagnostic checks             Updates statuspage, executives,
  and deployment rollbacks               and customer support channels
```

---

## 2. Severity Classification Matrix

| Severity | Customer Impact | MTTD Target | MTTR Target | Action Required |
| --- | --- | --- | --- | --- |
| **SEV-1** | Critical system down (>25% users affected / financial loss) | < 5 mins | < 30 mins | Immediate all-hands bridge, executive notification every 30m |
| **SEV-2** | Significant degradation / core feature failing | < 15 mins | < 2 hours | Dedicated Slack channel + SRE bridge |
| **SEV-3** | Minor non-blocking issue or single customer affected | < 1 hour | < 1 business day | Standard ticket queue triage |

---

## 3. Blameless Post-Mortem Template

```markdown
# Incident Post-Mortem: [YYYY-MM-DD] [Incident Title]

## Executive Summary
- **Impact Duration**: 42 minutes (14:12 UTC to 14:54 UTC)
- **User Impact**: ~14,200 checkout attempts failed with HTTP 504.
- **Root Cause**: Database connection pool exhaustion caused by missing query index during peak traffic burst.

## Timeline (UTC)
- **14:12** - Latency alert fires for `checkout-service`.
- **14:15** - Incident declared SEV-1; Incident Commander appointed.
- **14:28** - Operations lead discovers 100% DB thread pool saturation.
- **14:38** - Hotfix index deployed to RDS reader/writer.
- **14:54** - Latency and error rates return to nominal SLO levels.

## Lessons Learned & Action Items
| Action Item | Type | Owner | Priority | Target Date |
|---|---|---|---|---|
| Implement query timeout limit on checkout repo | Preventative | @alice | P1 | Next Sprint |
| Add DB connection pool saturation SLO alert | Detection | @bob | P1 | 3 Days |
```

---

## 4. Scribe Discipline & Role Handoff

**The Scribe is not optional.** Without one, the timeline is reconstructed afterwards from
memory and Slack scroll-back, which is exactly when detail is lost and blame appears.

The Scribe records, in the incident channel, with timestamps in UTC:

- every state change ("error rate back under 1%"), not commentary;
- every action taken and by whom, including the ones that did not work;
- every hypothesis raised and how it was ruled in or out;
- every decision the Incident Commander makes, and the time it was made.

That log is the post-mortem's timeline section, written for free during the incident.

**Explicit handoff.** Fatigue causes second incidents. Every role hands over out loud and in
writing after ~2 hours or at a shift boundary:

```text
HANDOFF 14:05Z — Incident Commander: @alex -> @sam
Current state: SEV2, error rate 4% (was 18%), degraded checkout only.
Working theory: connection-pool exhaustion after the 12:40Z deploy.
In flight: @dana rolling back to build 4471 (ETA 14:15Z).
Ruled out: DNS, upstream provider, node pressure.
Next decision point: 14:20Z — if rollback does not clear it, escalate to SEV1 and page DBRE.
```

The outgoing IC states it, the incoming IC confirms in channel, and the Scribe logs both. No
silent handoffs, and never more than one Incident Commander at a time.