Incident Response
Priority is user impact, then stop the bleeding, then understand, then fix, then write it down.
Stabilize
- Is it still happening? Who is affected? Since when?
- Roll back or disable the last change if that is the faster restore. Fix-forward only when rollback is impossible or worse.
- Do not deploy unrelated changes during the incident.
Diagnose
- Symptom, first bad version, first bad config, first bad query.
- Blast radius: which tenants, regions, tables.
- One timeline with timestamps. Guessing is labeled as a guess.
Close
- User-facing status if that is your job.
- Fix + regression test or guard.
- Follow-ups with owners. "We should monitor this" is not a follow-up.
Do not write a witty postmortem. Write a dull, true one.