Incident Response

Incident Response

Your objective is a shorter outage, not a correct theory. Mitigation comes before diagnosis, every time.

1. Declare and set severity

Say it out loud so everyone shares one picture.

Unsure between two levels? Take the higher one. Downgrading is cheap; upgrading late is not.

Escalate to a human immediately for suspected data loss, a suspected security breach, customer data exposure, legal or regulatory exposure, or a decision to accept prolonged downtime. Those are not technical calls.

2. What changed

Before theorising, check the ten minutes before the first alert: a deploy, a feature flag, a config push, a certificate, a dependency, a traffic shift, a scheduled job. This explains most incidents and takes two minutes to check.

3. Mitigate

Roll back, flip the flag, shed load, fail over, or degrade the feature. You do not need to know why in order to stop the bleeding.

Announce every action before taking it: what you are doing, what it should do, and the signal that will confirm it. An untracked change during an incident creates a second incident inside the first.

Then verify with telemetry, quoting the metric before and after. Not with hope.

4. Communicate on a clock

Every 30 minutes for Sev 1 and 2, even with no news. "Still investigating, next update at 14:30" is a valid update and prevents six people from interrupting the responders.

[SEV<n>] <one-line impact: who cannot do what>
Started: <time> · Status: investigating | mitigating | monitoring | resolved
Impact: <scope, number of users if known>
Current action: <what is happening right now>
Next update: <time>

5. Preserve evidence, then diagnose

Only after impact has stopped. Before anything is cleaned up or restarted, capture: logs for the window, a metrics snapshot, the exact deployed version, and a copy of any bad state. Then use systematic-debugging.

6. Postmortem, within 48 hours

# Postmortem: <title>
- **Impact:** <who, what they could not do, how long, how many>
- **Severity:** <n>
- **Detected by:** <alert / customer / engineer> — <how long after onset>

## Timeline
<UTC: change, onset, detection, mitigation, resolution>

## What happened
<the mechanism, in plain language>

## Why it was not caught
<the gap in tests, review, staging, or alerting>

## Why detection took <n> minutes
<a separate question from the cause, and usually the higher-leverage one>

## Action items
| Action | Owner | Due | Prevents recurrence / reduces detection time |

Rules:

• No names as causes. "Someone made a mistake" is never a root cause; the system that let the mistake reach production is.
• Detection time gets its own section. Reducing it usually pays more than preventing one specific cause, and it is almost always skipped.
• Every action item has an owner and a date, or it is not an action item.

Verification

• Impact stopped before diagnosis started, or you state why mitigation was impossible.
• Every mitigation has a quoted before-and-after metric.
• Updates went out on the clock.
• Evidence was preserved before cleanup.
• The postmortem separates cause from detection and names no person as a cause.

Red flags