Incident Response

Follow this playbook for every incident. No shortcuts.

The 5 Steps

1. CONTAIN (first 5 minutes)

Stop the bleeding. Prevent further damage.

• Can we rollback to the last working version?
• Can we disable the broken feature without taking down everything?
• Is customer data at risk?
• Who needs to be notified right now?

2. DIAGNOSE (next 15 minutes)

Find the root cause using data, not guesses.

• Check logs: what changed? when did errors start?
• Check deployments: was there a recent deploy?
• Check external dependencies: is a third-party service down?
• Check metrics: traffic spike? memory leak? disk full?
• Form a hypothesis and verify before acting

3. FIX (next 30 minutes)

Apply the minimum change needed.

• Prefer reverting over forward-fixing
• If forward-fixing: change one thing, verify, repeat
• Do NOT refactor while fixing an incident
• Do NOT add new features while fixing an incident

4. VERIFY (next 15 minutes)

Confirm the fix works and has no side effects.

• Is the error gone from logs?
• Are affected customers able to use the service?
• Are metrics back to normal baselines?
• Run a smoke test on the critical path

5. POSTMORTEM (within 24 hours)

Document what happened so it doesn't happen again.

• Timeline — minute-by-minute of the incident
• Root cause — what actually broke and why
• Impact — who was affected, for how long, revenue impact
• What went well — what worked in the response
• What went poorly — where we lost time
• Action items — specific preventive measures with owners and dates

Severity Classification

Communication Template

[STATUS: Investigating/Identified/Monitoring/Resolved]
Impact: [what's affected]
Started: [time]
Current action: [what we're doing right now]
Next update: [when]