Incident Response Runbook

When to Use

• Use when: a production incident is declared or suspected
• Use when: writing an on-call runbook for a new service
• Use when: generating a PIR / postmortem after an incident closes
• Use when: improving incident response process after a painful incident
• Do NOT use for: proactive SLO/SLI design — use slo-design
• Do NOT use for: security incident response — use red-team-check / blue-team-fix

Severity Levels

Agree on severity definitions before incidents happen. Typical classification:

Key rule: SEV-1 and SEV-2 are declared by the on-call engineer, not management. Never let process delay a SEV-1 declaration.

Incident Roles

Single person can cover multiple roles on small teams. IC role must be explicit — "who's IC?" is the first question to answer.

Incident Response Checklist

Phase 1 — Declare and orient (0–5 min)

□ Declare severity (SEV-1/2/3) — don't wait for certainty
□ Open incident channel (#inc-YYYY-MM-DD-short-name)
□ Assign IC, TL, Scribe
□ Post initial status: "We are investigating X. Impact: Y. IC: @name"
□ Set update cadence: SEV-1 = every 10 min; SEV-2 = every 30 min

Phase 2 — Triage (5–30 min)

□ What is broken? (symptom, not cause)
□ Who is affected? (% users, regions, tenants)
□ When did it start? (first alert vs. first user report)
□ What changed? (deploys, config, traffic spike in last 2 hours)
□ Is there a mitigation? (rollback, feature flag, traffic shift)
□ Apply mitigation BEFORE root cause analysis — stability first

Phase 3 — Resolve

□ Mitigation confirmed working (SLI recovering toward SLO)
□ Monitor for 15–30 min before declaring resolved
□ Post "Resolved" in incident channel with timeline
□ Update status page
□ Schedule PIR within 5 business days (SEV-1) or 10 days (SEV-2)

Communication Templates

Internal status update (every 10–30 min)

[SEV-1 Update — 14:35 UTC]
Status: Investigating
Impact: ~30% of API requests failing with 503 (US-East region)
What we know: Spike started at 14:10 UTC following config deploy at 14:08
What we're doing: Rolled back config. Monitoring recovery.
Next update: 14:45 UTC
IC: @yana

External status page update

Investigating elevated error rates
We are investigating reports of increased error rates affecting [product].
Engineers are actively working on a fix. Next update in 30 minutes.
Started: 14:10 UTC

Keep external updates factual, calm, and jargon-free. Never speculate publicly.

Post-Incident Review (PIR) Template

Generate within 5 days of SEV-1, 10 days of SEV-2.

## Incident: [Short title]
**Date:** YYYY-MM-DD  **Severity:** SEV-X  **Duration:** Xh Xm
**IC:** @name  **TL:** @name

## Summary
[2–3 sentences: what happened, who was affected, what fixed it]

## Impact
- Users affected: ~X (X%)
- Duration: Xh Xm
- Error budget consumed: X minutes (X% of monthly budget)

## Timeline
| Time (UTC) | Event |
|---|---|
| HH:MM | First alert fired |
| HH:MM | IC declared SEV-X |
| HH:MM | Root cause identified: [X] |
| HH:MM | Mitigation applied |
| HH:MM | Service recovered |
| HH:MM | Incident closed |

## Root Cause
[Technical explanation — what caused the issue. Avoid blame. Focus on system.]

## Contributing Factors
- [Factor 1]
- [Factor 2]

## What Went Well
- [Response was fast because...]
- [Alert fired within X minutes]

## What Went Poorly
- [Alert threshold too high — missed first 20 min]
- [Runbook out of date]

## Action Items
| Action | Owner | Due | Priority |
|---|---|---|---|
| Update runbook for X scenario | @name | YYYY-MM-DD | High |
| Lower alert threshold for Y | @name | YYYY-MM-DD | Medium |

Blameless Culture Rules

• PIR is about the system, not the person who deployed the change.
• "Human error" is never a root cause — it is a symptom of missing safeguards.
• The real question: why did the system allow this failure mode to reach production?
• Public PIRs are a team-level document. Strip individual names from public versions.

Anti-Fake-Pass Rules

Before claiming "incident response is set up", you MUST show:

• Severity levels defined and agreed (not just "we know what SEV-1 means")
• IC role defined and someone assigned in every incident
• On-call runbook exists for at least the primary failure modes
• PIR process: who writes it, when, where it's stored
• Action items from last PIR tracked to completion (not just written)

Reference: gates/anti-fake-pass-gate.md