Postmortem

/postmortem -- Single-Incident Postmortem

Fast-path companion to Pillar 3 (Achieve Stability). Writes a single-incident postmortem right after a production incident, instead of waiting for a full /stability audit. The book's bar: every business-impacting outage gets a full, executive-visible investigation -- not a quiet fix.

When to invoke

• Immediately after a production incident is resolved, while details are still fresh
• When a Sev-1/Sev-2 incident closed with "fixed it" and no written record
• As the required output of an incident retro, before the team moves on

What it checks

1. Timeline -- detection time, response time, mitigation time, full resolution time -- in that order, with gaps between them called out
2. Root cause -- the actual mechanism that caused the incident, not just the symptom that was patched
3. Blast radius -- what broke, for whom, for how long, and any data or revenue impact
4. Contributing factors -- process gaps that let this happen (missing test, missing alert, missing runbook, single point of failure)
5. Action items -- concrete, owned, dated follow-ups -- not just "be more careful"

Inputs

• Incident record -- the alert/page that fired, chat logs from the incident channel, or the user's own account of what happened and when
• Fix -- the PR, config change, or manual action that resolved it
• Prior history -- check whether this same failure mode has occurred before (search incidents//postmortems/ if the repo has one, or ask); a repeat incident is itself a finding
• If timestamps are fuzzy, say so rather than inventing precision -- "approximately" is fine, a fabricated timestamp is not

Example output

Postmortem -- DB connection pool exhaustion, Atlas API, 2026-07-14

Timeline:

Time	Event
14:02	Alert fires: p99 latency > 5s
14:06	On-call engineer acknowledges
14:19	Root cause identified: connection pool exhausted by a retry loop in the billing webhook handler
14:24	Mitigated: pool size increased, retry loop disabled
15:10	Fully resolved: fix deployed, retry loop rewritten with backoff

Root cause: A billing webhook retry loop had no backoff and no cap, so a single slow downstream call multiplied into hundreds of held connections within minutes.

Blast radius: ~12 minutes of degraded API latency (p99 5-8s) for all customers; no data loss; no failed payments (webhook itself is idempotent).

Contributing factors: No max-retry limit on the webhook handler; no alert on connection-pool saturation specifically (the p99-latency alert caught it indirectly, 4 minutes after saturation started). This is the second connection-pool incident in 90 days -- the first (2026-04-22) was fixed ad hoc with no postmortem, so this contributing factor was never addressed.

Action items:

1. Add exponential backoff + max-retry cap to the webhook handler -- @owner, this week
2. Add a dedicated connection-pool-saturation alert (not just downstream latency) -- @owner, this sprint
3. Write a runbook for "connection pool exhausted" now that it's recurred twice -- @owner, this sprint

Reference

See ../../docs/FIVE_PILLARS.md and ../stability/SKILL.md for the full production-readiness audit this feeds into.