Engineering Incident Postmortem

Incident post-mortem

Purpose

Post-mortems fail in three predictable ways: they stop at the trigger ("a bad deploy went out") and call it a cause, they blame a person instead of the conditions that made the error easy, and they end in actions nobody owns. This skill fixes the timeline to evidence, forces the cause analysis past the first plausible answer, and refuses to close on an action without an owner and a date.

Prerequisites

• Inputs: the incident record (ID, severity, declared and resolved times); the alert or report that started detection; deploy/change log for the affected services covering the 48h before impact; chat transcript or timeline of responder actions; the customer-visible impact (what broke, for whom, for how long, measured — not estimated).
• Access: monitoring/dashboards for the impact window, change management records, ticket system to file actions.

If the impact window or the change log is missing, say so and stop. A post-mortem built on recollection produces a plausible story and the wrong actions.

Procedure

1. Fix the timeline before writing any narrative. Build a table of timestamped events with a source for each. Every row is one of: a change, a signal, a human action, or a state transition. If a row has no source, mark it [unverified] — never silently promote recollection to fact.

   Time (UTC)	Event	Type	Source
   09:14	Config change deployed to payments-api	change	deploy log
   09:41	Error rate crosses 2%	signal	dashboard
   09:58	On-call paged	signal	alert record

2. Derive the four durations. Time to detect (impact start → first signal), time to page (signal → human engaged), time to diagnose (engaged → cause identified), time to mitigate (diagnose → impact ends). Whichever is longest is where the actions must concentrate. Teams routinely write five actions about the code change and none about the 44 minutes of undetected impact.

3. Separate trigger, cause, and contributing conditions.

   Layer	Question it answers	Test that it is the right layer
   Trigger	What set it off this time?	Would removing it prevent this recurrence only?
   Cause	What made that trigger able to cause harm?	Would fixing it stop a class of incidents?
   Contributing conditions	What made detection or recovery slow?	Would fixing it shorten the next unrelated incident?

   A trigger is never a sufficient answer. "Someone deployed a bad config" is a trigger; "config changes deploy to all regions simultaneously with no validation gate" is the cause.

4. Push each cause chain until it lands on something you can change. Stop when the next "why" would be a statement about human nature or organisational budget you cannot action. Stopping early gives you "the engineer made a mistake"; going too far gives you "we are under-resourced". The actionable band sits between: the missing guardrail, the misleading interface, the alert that did not exist.

5. Test each candidate cause against the counterfactual. For each, ask: if this had been different, would the impact have been prevented or materially reduced? If no, it is context, not cause — move it to the narrative and drop it from the action list.

6. Write actions in the required shape. Every action: a verb, a specific system, an owner (a named individual, never a team), a due date, and a ticket ID. Classify each:

   Class	Reduces	Ship within
   Prevent	Probability of recurrence	The current cycle
   Detect	Time to detect	The current cycle
   Mitigate	Blast radius or time to recover	Next cycle
   Process/knowledge	Repeat confusion in response	Next cycle

   If every action is class Prevent, the review is incomplete — detection and mitigation gaps almost always exist and are usually cheaper to close.

7. Run the blameless check before circulation. Scan the draft for named individuals. A person's name is acceptable in the timeline as an actor ("on-call acknowledged") and unacceptable in the cause section. Rewrite "X misconfigured the limit" as "the limit could be set to an invalid value without validation".

8. Set a verification date. Actions are marked done when the change is in production, not when the ticket closes. Book a check 30 days out that confirms the guardrail exists and fires.

Failure modes this skill exists to prevent

• Cause laundering. The trigger is restated in passive voice and presented as the root cause. Caught by step 3's "class of incidents" test.
• Action inflation. Fourteen actions, none prioritised; six months later two are done and none are the important ones. Cap at five; the rest are backlog.
• Duration blindness. The whole review addresses the change and ignores that detection took 44 minutes. Caught by step 2.
• Silent unknowns. Gaps in the timeline get smoothed over by narrative. Write "we do not know why the retry storm began; investigation ticket X" — an explicit unknown is a finding, an implicit one is a lie.

Data handling

Classification: Internal by default; Confidential if the incident involved a security breach, personal data exposure, or financial loss. Post-mortem text must not carry customer identifiers, account numbers, or log lines containing personal data — reference the affected records by count and by ticket, not by value. If someone pastes raw customer records or credentials into the review, flag it and do not incorporate them.

Boundaries

• Incident still active, or you are deciding mitigations in the moment — use engineering-incident-command, not this.
• Change was routine and had no customer or availability impact — record it in the ticket; a post-mortem adds process cost and no learning.
• Security breach requiring regulatory notification — this skill covers the engineering review only; notification obligations run through legal-compliance-breach-notification.
• Choosing between architectural options to close an action — that decision belongs in engineering-decision-record.

Hand-offs

• Receives from: engineering-incident-command (timeline, severity, responder log).
• Routes to: engineering-decision-record when a remediation implies an architectural change; it-change-management when the fix needs a controlled change window; engineering-code-review for the review depth on the fix itself.
• See also: data-analytics-report-qa when impact numbers must be quoted externally and need a verification pass first.