Incident Response And Postmortems

Incident Response and Postmortems

Overview

Define how the team detects, mitigates, communicates about, and learns from production incidents — before the first one. Output is .forge/incident-response.md — severity definitions with response times, the declare-mitigate-communicate-resolve-review flow, runbook template, blameless postmortem template, and the comms templates (internal and external). Pairs with observability (alerts feed the declare step) and error-handling-and-resilience (the failure modes alerts watch for).

When to Use

• A service is going to production and there's no on-call process
• An incident just happened and the team handled it ad-hoc — write the process before the next one
• A postmortem is due and there's no template
• Severity levels are inconsistent across teams
• A runbook is needed for a critical service

When NOT to Use

• Localhost-only or pre-prod systems with no users
• A trivial bug fix — that's debugging-and-recovery
• A near-miss that didn't affect users — log it, learn from it, but don't fire the full process

Common Rationalizations

Red Flags

• No documented severity definitions
• A critical service with no runbook
• A postmortem that uses words like "negligence", "should have known", "failure to follow procedure" — blame language
• Action items in a postmortem without an owner and a deadline
• The same incident class recurring (auth outage, queue backup, DB lock) without an action item from the previous postmortem closed
• No external comms during a customer-visible outage
• The on-call hand-off has no documented context (no runbook, no recent incident summary)

Core Process

Step 1: Define severity levels

In .forge/incident-response.md:

Step 2: Write the response flow

Five steps, in order:

1. Declare — anyone can declare. Use the standard channel (#incident-active). State: severity guess, what's broken, what's the impact.
2. Mitigate — restore service first. Root-cause investigation comes after. Acceptable mitigations: rollback, feature flag off, traffic shift, scale up.
3. Communicate — internal updates every 30min for Sev1, every 60min for Sev2. External status page updated within 15min of declare (Sev1) or 30min (Sev2). Comms owner is named, not "whoever notices."
4. Resolve — service restored, customer-visible impact ended. Mark resolved on status page.
5. Review — postmortem within 5 business days for Sev1/Sev2.

Step 3: Create the runbook template

For every critical service (anything with a Sev1 or Sev2 potential), a runbook must contain:

• Service summary — what it does, what fails when it goes down
• Owners — primary on-call, escalation chain
• Dashboards — links to the observability dashboards (cross-ref observability skill)
• Common failure modes — top 5 with first-response steps
• Mitigation playbook — feature flag locations, rollback commands, traffic shift recipes
• Escalation criteria — when to page the next tier
• Recovery verification — how you know it's actually fixed (synthetic check, key metric returning to baseline)

Cross-reference the runbook URL in every alert (cross-ref observability rule: every alert has a runbook link).

Step 4: Write the blameless postmortem template

In .forge/incident-response.md:

# Postmortem: <one-line summary>

## Impact
- Severity: Sev1/2/3
- Duration: HH:MM start to HH:MM end (TZ)
- Affected: <number of users, % of traffic, $ if measurable>

## Timeline (UTC, all times verified from logs)
- 14:02 — Deploy v2.4.1 to production
- 14:08 — Error rate climbs above 5% on /api/checkout
- 14:11 — Alert fires; on-call paged
- 14:14 — On-call acks
- 14:22 — Mitigation: rollback to v2.4.0 initiated
- 14:26 — Rollback complete, error rate returns to baseline
- 14:31 — Incident resolved; status page updated

## Root cause
<what happened, mechanically. No blame language.>

## Contributing factors
- <missing alert, weak test, flaky dependency, slow rollback tooling, etc.>

## What went well
- <fast detection, clean rollback, good comms>

## What we learned
- <pattern that future work should respect>

## Action items
| ID | Action | Owner | Deadline | Tracking |
|---|---|---|---|---|
| AI-1 | Add integration test covering the checkout path | @owner | YYYY-MM-DD | LINK-123 |
| AI-2 | Reduce rollback time from 8min to <2min | @owner | YYYY-MM-DD | LINK-124 |

Blameless rule: the word "should have" is forbidden. Replace with "the system did not have." Every action item targets a system, not a person.

Step 5: Define comms templates

Three audiences, three templates:

• Internal status update (every 30min Sev1 / 60min Sev2 in #incident-active): current status, mitigation in progress, ETA, next update time
• External status page (15min Sev1 / 30min Sev2): what's affected, what we're doing, when the next update is — no internal jargon
• Customer comms post-resolution (within 24h for Sev1, 72h for Sev2): impact summary, root cause in plain language, what we're doing to prevent recurrence

Verification

• .forge/incident-response.md written
• Severity levels defined with response-time SLAs
• Every critical service has a runbook linked from its alerts
• Postmortem template includes timeline, impact, root cause, contributing factors, action items with owners
• Last 3 Sev1/Sev2 postmortems completed within 5 business days
• No postmortem in the last 90 days contains "should have" or blame language
• Every action item from the last 5 postmortems has owner + deadline (open or closed)
• External status page updated within SLA on the last 3 customer-visible incidents