# Incident Postmortem > Write blameless incident postmortems with timeline reconstruction, root cause analysis, action items, and preventive measures - Skill: `w95/incident-postmortem` (Agent Skill) - Install (CLI): `npx skillmds@latest add w95/incident-postmortem` - Raw SKILL.md: https://api.skillmd.com/api/skills/w95/incident-postmortem/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: w95 (https://skillmd.com/u/w95) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/w95/incident-postmortem --- # Incident Postmortem Builder ## Overview Create blameless incident postmortems that transform operational disruptions into learning opportunities. These documents focus on system failures and process gaps, not individual blame, enabling continuous improvement and preventing recurrence. ## Core Principles 1. **Blameless**: Focus on systems, not people. "Why did this happen?" not "Who screwed up?" 2. **Psychological Safety**: Team members must feel safe discussing root causes without fear 3. **Data-Driven**: Base findings on logs, metrics, and facts; not assumptions 4. **Action-Oriented**: Every finding leads to actionable improvements 5. **Learning Culture**: Treat incidents as valuable learning events, not failures 6. **Transparency**: Share findings broadly; communicate changes to prevent similar incidents ## Timeline Reconstruction Create a detailed chronology of events: ``` Time (UTC) | Who | What | Evidence | Context -----------|-----|------|----------|---------- 2024-02-15 14:32 | Jenkins | Deploy v2.1.3 (buggy) | Logs | Automated Friday deploy 14:35 | Customer | Website errors | CloudFront | 500 errors reported 14:37 | On-call | PagerDuty alert | Alert | Error rate exceeded threshold 14:42 | Eng team | Investigation starts | Slack #incidents | Identified deploy cause 14:55 | Lead | Rollback initiated | Logs | Reverted to v2.1.2 15:02 | On-call | Error rate normal | Metrics | Customers back to normal 15:30 | Team | Root cause meeting | Notes | Identified root cause ``` **Timeline Template**: - **T+0 (Alert)**: When first detected - **T+X (Detection)**: When incident was recognized - **T+Y (Communication)**: When stakeholders notified - **T+Z (Mitigation)**: When incident owner took action - **T+N (Resolution)**: When system returned to normal - **Duration**: Total time from detection to resolution ## Root Cause Analysis (5 Whys) Go beyond the obvious cause to find systemic issues: ``` Incident: Website down for 28 minutes Why 1: Why did website go down? Answer: Deployment v2.1.3 contained a bug causing infinite loop in auth service Why 2: Why did the bug reach production? Answer: Code review missed the issue; test suite didn't catch it Why 3: Why didn't test suite catch the infinite loop? Answer: Load/stress tests only run occasionally; not part of standard CI pipeline Why 4: Why aren't load tests mandatory in CI? Answer: Historically slow; team prioritized speed over reliability Why 5: Why does team optimize for deploy speed over testing? Answer: Pressure to ship features fast; no documented standard for testing rigor ROOT CAUSE: Process gap - no mandatory load testing in CI; pressure to ship ``` **Avoid**: - Stopping too early ("operator didn't notice error") - Human error as root cause ("developer made a mistake") - Unclear systemic issues **Focus on**: - Process failures - Monitoring gaps - Communication breakdowns - Knowledge gaps - Tool limitations - Architectural weaknesses ## Contributing Factors (Swiss Cheese Model) Most incidents involve multiple failures aligning: ``` Incident: Late-night 30-minute outage Contributing Factors: 1. Code change made Friday afternoon (rush to deploy before weekend) 2. No automated rollback capability (manual process) 3. On-call engineer had weak knowledge of new code (hired 3 weeks ago) 4. No load test coverage for auth service changes (technical debt) 5. Monitoring alert threshold set too high (missed early warning) 6. Deployment not staged; went straight to production (process gap) 7. No change advisory board approval (governance gap) Any ONE of these alone wouldn't have caused incident. Combined: 28-minute outage. ``` ## Root Cause vs. Proximate Cause **Proximate Cause** (immediate cause): - Infinite loop in authentication code - Deployment that shouldn't have happened - Missing monitor alert **Root Cause** (systemic failure): - Code review process insufficient for critical changes - Deployment process lacked staged/canary deployment - Testing strategy doesn't include stress tests - Knowledge gap in on-call team for recent changes Focus postmortem on root causes, not proximate causes. ## Action Items (Follow-up) **Structure**: [Priority] | [What] | [Why] | [Owner] | [Due Date] | [Status] ### Immediate Actions (0-7 days) ``` CRITICAL | Deploy hotfix for infinite loop | Prevent recurrence | Sarah | 2024-02-15 | DONE HIGH | Document code change impact | Knowledge transfer | John | 2024-02-16 | IN PROGRESS MEDIUM | Post-incident communication to customers | Transparency | PM | 2024-02-15 | DONE ``` ### Short-term Actions (1-4 weeks) ``` HIGH | Implement automatic canary deployment | Catch issues pre-production | DevOps | 2024-03-01 | PENDING HIGH | Add auth load tests to CI pipeline | Catch performance issues early | QA | 2024-03-01 | PENDING MEDIUM | Onboard new on-call engineer on recent changes | Knowledge gap closure | Tech Lead | 2024-02-28 | IN PROGRESS ``` ### Long-term Actions (1-3 months) ``` MEDIUM | Implement automated rollback capability | Faster recovery time | Arch | 2024-04-15 | PENDING LOW | Review change advisory board process | Governance improvement | Ops | 2024-05-01 | PENDING LOW | Schedule quarterly load testing for critical services | Proactive risk management | Perf | 2024-06-01 | PENDING ``` **SMART Action Items**: - **S**pecific: What exactly needs to be done? - **M**easurable: How will we know it's complete? - **A**ssignable: Who owns this? - **R**ealistic: Can it actually be done? - **T**ime-bound: When is it due? ## Preventive Measures **Prevention Strategy**: How do we prevent this type of incident? 1. **Process Changes**: - Implement staged/canary deployments for critical services - Add code review requirement for auth service changes - Require load test passing for critical path changes 2. **Monitoring & Alerting**: - Lower error rate alert threshold (early warning) - Add CPU/memory alerts for auth service - Add canary endpoint synthetic monitoring 3. **Automation**: - Automatic rollback if error rate >2% - Load test gate in CI pipeline (mandatory, not optional) - Automated chaos engineering tests weekly 4. **Documentation & Training**: - Document architecture of auth service - Create runbook for auth service incidents - Schedule knowledge transfer session for on-call team 5. **Organizational**: - Remove deadline pressure; don't deploy Friday afternoon - Add on-call engineer to code reviews of critical services - Establish incident SLA: detection to resolution <15 minutes for P0 incidents ## Template Structure **INCIDENT POSTMORTEM** Incident ID: INC-2024-047 Date: 2024-02-15 Duration: 28 minutes (14:35 UTC - 15:03 UTC) Impact: Website unavailable; 0.5M page requests failed Severity: P1 (Critical) **INCIDENT SUMMARY** [1 paragraph overview of what happened and impact] **TIMELINE** [Chronological events table] **ROOT CAUSE ANALYSIS** Primary Root Cause: [High-level finding] 5 Whys Analysis: [Why chain] Contributing Factors: [List of systemic issues] **IMPACT ANALYSIS** - Customers Affected: [Number or percentage] - Duration: [Minutes] - Revenue Impact: [If quantifiable] - Reputation Impact: [Qualitative assessment] - Data Loss: [Yes/No, details if yes] **DETECTION & RESPONSE** - Detection Time: [Minutes to detect] - Response Time: [Minutes to start mitigation] - Resolution Time: [Minutes to full recovery] - Response Quality: [Smooth/Some delays/Chaotic - why?] **WHAT WENT WELL** - [Good thing 1]: Enabled [outcome] - [Good thing 2]: Enabled [outcome] - [Good thing 3]: Enabled [outcome] *Recognize excellent work; reinforce good behaviors* **WHAT COULD BE BETTER** - [Gap 1]: Impact was [consequence] - [Gap 2]: Impact was [consequence] - [Gap 3]: Impact was [consequence] **ACTION ITEMS** [Immediate, short-term, long-term actions with owners and dates] **PREVENTIVE MEASURES** [How we prevent this incident class in future] **APPENDICES** - Error logs (anonymized) - Customer communication - Monitoring graphs during incident - Architecture diagram - Related incidents (historical patterns) ## Postmortem Facilitation **Blameless Meeting Principles**: 1. **Start with context, not blame**: "Here's what was happening at 14:30 UTC..." 2. **Use neutral language**: "Code changed" not "Code was broken" 3. **Ask curious questions**: "What were you seeing on your screen?" not "Why didn't you check the logs?" 4. **Encourage storytelling**: Let people describe their experience; narrative flow 5. **Capture assumptions**: "I assumed..." statements reveal knowledge gaps 6. **No hierarchy**: On-call engineer's observations valued same as CTO's 7. **Record decisions**: Why did we choose rollback vs. fix? Document the thinking 8. **Record learnings**: What surprised people? What did they learn? **Participants**: - On-call engineer (incident responder) - Service owner - DevOps/Infrastructure team - Product/Business owner - Facilitator (experienced, neutral party) **Meeting Duration**: 30-60 minutes maximum **When to Hold**: Within 48 hours of incident resolution (while details are fresh) ## Distribution & Follow-up 1. **Share Widely**: Postmortem is internal tool for learning; share with full engineering org 2. **Executive Summary**: One-page summary for leadership 3. **Customer Communication**: Transparency about what happened and prevention measures 4. **Process Review**: Monthly review of open action items 5. **Trend Analysis**: Quarterly review - are we preventing incident classes or just firefighting? ## Preventing Similar Incidents **Incident Class Tracking**: - Authentication failures - Database performance degradation - Memory leaks - Configuration errors - Dependency failures - Deployment failures If same class happens twice: escalate prevention measures If same incident happens three times: organizational escalation (management review) --- **Use this skill to**: Transform incidents into learning opportunities, improve system resilience, and build a psychologically safe incident response culture.