dkenge123-oss Incident Postmortem
A postmortem exists to make the same failure impossible, not to assign fault. If it reads like a confession, it will teach no one.
Protocol
- Build the timeline from evidence. Logs, alerts, deploy history, chat transcripts. Record what happened and when in objective terms: detection time, escalation, mitigation, resolution. Reconstruct from artifacts, not memory.
- State impact concretely. Who was affected, for how long, and how badly. "Checkout was down for 23 minutes affecting an estimated N users" beats "there was an issue".
- Find the root cause, then keep asking why. The first cause is rarely the real one. "The server ran out of memory" — why? "A query returned unbounded results" — why? "No limit was enforced because the API contract assumed a filter" — there it is. Stop when the answer is a process or design gap, not a person.
- Separate the trigger from the vulnerability. The trigger is what happened this time; the vulnerability is why the system was fragile enough for it to matter. Fixing only the trigger leaves the next trigger free to exploit the same weakness.
- Write action items that are specific, owned, and dated. "Add a row limit to the report query (owner: X, by: date)" is an action item. "Be more careful with queries" is a wish. Each item should measurably reduce the chance or blast radius of recurrence.
- Keep it blameless. Describe decisions in terms of the information available at the time. People acted reasonably given what they knew; the system let a reasonable action cause harm. That framing is what makes people share the real story.
Never
- Never name individuals as causes. Name the gap that let a human error become an incident.
- Never close with "we'll be more careful" or any action item without an owner and a date.
- Never skip the postmortem because the fix was quick; the near-miss is the cheapest lesson you will get.
Done means
A reader who was not there understands what happened, why the system allowed it, and what specific, owned changes will stop it from happening again.