# Post Mortem > Guides a blameless post-mortem process for incidents, covering templates, root-cause analysis, action items, and prevention. - Skill: `chimeranext/post-mortem` (Agent Skill) - Install (CLI): `npx skillmds@latest add chimeranext/post-mortem` - Raw SKILL.md: https://api.skillmd.com/api/skills/chimeranext/post-mortem/raw - Safety review: PASS (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra, Coding & Dev Tools, Monitoring & Observability - Tags: Blameless, Incident Management, Incident Review, Post Mortem, Root Cause Analysis, Sre - Author: chimeranext (https://skillmd.com/u/chimeranext) - Updated: 2026-08-22 - Page: https://skillmd.com/skills/chimeranext/post-mortem --- # 📋 Skill: Post-Mortem Process ## 📋 Metadata | Atributo | Valor | |----------|-------| | **ID** | `sre-post-mortem` | | **Nivel** | 🔴 Avanzado | | **Versión** | 1.0.0 | | **Keywords** | `post-mortem`, `postmortem`, `incident-review`, `blameless`, `lessons-learned`, `root-cause-analysis` | | **Referencia** | [Google SRE - Postmortem Culture](https://sre.google/workbook/postmortem-culture/), [Blameless Postmortems](https://www.blameless.com/blog/blameless-postmortem) | ## 🔑 Keywords para Invocación - `post-mortem` - `postmortem` - `incident-review` - `blameless` - `lessons-learned` - `root-cause-analysis` - `incident-analysis` - `@skill:post-mortem` ### Ejemplos de Prompts ``` Crea un post-mortem para el incidente de ayer ``` ``` Implementa proceso de post-mortem blameless ``` ``` Genera template de post-mortem con timeline y action items ``` ``` @skill:post-mortem - Proceso completo de análisis de incidentes ``` ## 📖 Descripción Un post-mortem es un análisis estructurado y blameless de un incidente que busca entender qué pasó, por qué pasó, y cómo prevenir que vuelva a ocurrir. Este skill cubre la metodología completa de post-mortems, templates, mejores prácticas, y herramientas para facilitar el proceso. ### ✅ Cuándo Usar Este Skill - Después de incidentes críticos (SEV1, SEV2) - Cuando se consume error budget significativo - Incidentes que afectan múltiples usuarios - Incidentes que requieren escalación - Para cumplir con compliance requirements - Cuando hay patrones recurrentes de incidentes ### ❌ Cuándo NO Usar Este Skill - Incidentes menores sin impacto (SEV4) - Problemas resueltos en < 5 minutos sin impacto - Incidentes de mantenimiento planificados - Cuando no hay tiempo para análisis adecuado ## 🏗️ Proceso de Post-Mortem ``` ┌─────────────────┐ │ Incident Occurs │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Incident Resolved│ └────────┬────────┘ │ │ (Within 24-48 hours) ▼ ┌─────────────────┐ │ Schedule Meeting │ │ (All Stakeholders)│ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Collect Data │ │ - Logs │ │ - Metrics │ │ - Timeline │ │ - Testimonies │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Write Draft │ │ Post-Mortem │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Review Meeting │ │ (Blameless) │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Finalize & │ │ Publish │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Track Action │ │ Items │ └─────────────────┘ ``` ## 💻 Implementación ### 1. Post-Mortem Template ```markdown # Post-Mortem: [Incident Title] **Incident ID:** [INC-YYYYMMDD-HHMMSS] **Date:** [Date] **Duration:** [Start Time] - [End Time] (X minutes/hours) **Severity:** [SEV1/SEV2/SEV3] **Affected Services:** [Service names] **On-Call Engineer:** [Name] **Post-Mortem Lead:** [Name] **Status:** [Draft/Review/Final] --- ## Executive Summary [2-3 sentence summary for leadership. What happened, impact, and resolution.] **TL;DR:** [One sentence summary] --- ## Impact ### User Impact - **Users Affected:** [Number or percentage] - **Geographic Impact:** [Regions affected] - **Feature Impact:** [Which features were unavailable] ### Business Impact - **Revenue Impact:** [If applicable, estimated $] - **SLA Breach:** [Yes/No - which SLA] - **Error Budget Consumption:** [X% of monthly budget] - **Customer Complaints:** [Number] ### Technical Impact - **Services Down:** [List] - **Data Loss:** [Yes/No - details] - **Performance Degradation:** [Metrics] --- ## Timeline | Time (UTC) | Event | Actor | Notes | |------------|-------|-------|-------| | 10:00 AM | Alert fired: HighErrorRate | Prometheus | Error rate > 5% | | 10:02 AM | On-call engineer acknowledged | Engineer A | PagerDuty notification | | 10:05 AM | Initial investigation started | Engineer A | Checking logs | | 10:10 AM | Identified pattern in logs | Engineer A | Database connection errors | | 10:15 AM | Root cause identified | Engineer A | Connection pool exhaustion | | 10:20 AM | Mitigation deployed | Engineer B | Increased pool size | | 10:25 AM | Service restored | System | Metrics normalized | | 10:30 AM | Incident resolved | Engineer A | All systems healthy | **Key Milestones:** - **Detection Time:** 10:00 AM - **Response Time:** 2 minutes - **Time to Identify:** 15 minutes - **Time to Mitigate:** 20 minutes - **Time to Resolve:** 30 minutes - **Total Downtime:** 30 minutes --- ## Root Cause Analysis ### Primary Root Cause [Detailed explanation of what caused the incident. Use the "5 Whys" technique if applicable.] **What Happened:** [Description of the immediate cause] **Why It Happened:** [Underlying technical reason] **Why That Happened:** [Deeper systemic reason] ### Contributing Factors 1. **[Factor 1]** - **Impact:** [How it contributed] - **Why it existed:** [Systemic reason] 2. **[Factor 2]** - **Impact:** [How it contributed] - **Why it existed:** [Systemic reason] 3. **[Factor 3]** - **Impact:** [How it contributed] - **Why it existed:** [Systemic reason] ### Detection Gaps - [What we should have detected earlier] - [Why we didn't detect it] - [What monitoring/metrics were missing] --- ## Resolution ### Immediate Actions Taken 1. **[Action 1]** - **Who:** [Name] - **When:** [Time] - **Result:** [Outcome] 2. **[Action 2]** - **Who:** [Name] - **When:** [Time] - **Result:** [Outcome] ### Long-term Fix [What was done to permanently resolve the issue] --- ## Action Items ### Immediate (Next 24 hours) - [ ] **Item 1** - Owner: [Name] - Due: [Date] - Description: [Details] - [ ] **Item 2** - Owner: [Name] - Due: [Date] - Description: [Details] ### Short-term (Next week) - [ ] **Item 1** - Owner: [Name] - Due: [Date] - Description: [Details] - [ ] **Item 2** - Owner: [Name] - Due: [Date] - Description: [Details] ### Long-term (Next month) - [ ] **Item 1** - Owner: [Name] - Due: [Date] - Description: [Details] - [ ] **Item 2** - Owner: [Name] - Due: [Date] - Description: [Details] **Action Item Tracking:** - Total Items: [X] - Completed: [Y] - In Progress: [Z] - Not Started: [W] --- ## Lessons Learned ### What Went Well ✅ 1. **[Positive aspect 1]** - **Why it helped:** [Explanation] - **How to replicate:** [Actionable steps] 2. **[Positive aspect 2]** - **Why it helped:** [Explanation] - **How to replicate:** [Actionable steps] ### What Could Be Improved 🔄 1. **[Improvement 1]** - **Current state:** [What happened] - **Desired state:** [What should happen] - **How to improve:** [Actionable steps] 2. **[Improvement 2]** - **Current state:** [What happened] - **Desired state:** [What should happen] - **How to improve:** [Actionable steps] ### Surprises 🤔 - [Unexpected behavior or findings] - [Things we learned about the system] --- ## Prevention ### How to Prevent This Incident 1. **[Prevention measure 1]** - **Type:** [Monitoring/Process/Code/Infrastructure] - **Priority:** [High/Medium/Low] - **Effort:** [Small/Medium/Large] 2. **[Prevention measure 2]** - **Type:** [Monitoring/Process/Code/Infrastructure] - **Priority:** [High/Medium/Low] - **Effort:** [Small/Medium/Large] ### Similar Incidents - [Link to related post-mortems] - [Patterns identified] --- ## Follow-up ### Post-Mortem Review - **Scheduled Date:** [Date] - **Participants:** [Names] - **Status:** [Scheduled/Completed] ### Action Item Review - **Next Review Date:** [Date] - **Owner:** [Name] - **Last Updated:** [Date] ### Communication - **Internal:** [Who was notified] - **External:** [Customer communication if applicable] - **Status Updates:** [Where updates were posted] --- ## Metrics ### Incident Metrics - **MTTR (Mean Time To Resolution):** [X minutes] - **MTTD (Mean Time To Detection):** [X minutes] - **MTTI (Mean Time To Identify):** [X minutes] - **MTTM (Mean Time To Mitigate):** [X minutes] ### Historical Comparison - **Similar incidents in last 6 months:** [Number] - **Trend:** [Improving/Stable/Worsening] --- ## Appendix ### Logs & Evidence - [Links to relevant logs] - [Screenshots of dashboards] - [Metrics graphs] ### Related Documentation - [Runbooks used] - [Related post-mortems] - [Design documents] ### Team Feedback - [Quotes from team members] - [Additional context] --- **Document Version:** 1.0 **Last Updated:** [Date] **Next Review:** [Date] ``` ### 2. Post-Mortem Automation Script ```python # postmortem/generator.py from dataclasses import dataclass, field from datetime import datetime from typing import List, Optional, Dict from enum import Enum import json class IncidentSeverity(Enum): SEV1 = "critical" # Service down SEV2 = "high" # Major degradation SEV3 = "medium" # Minor issues SEV4 = "low" # Informational @dataclass class TimelineEvent: time: datetime event: str actor: str notes: Optional[str] = None @dataclass class ActionItem: id: str description: str owner: str due_date: datetime priority: str # immediate, short-term, long-term status: str = "not-started" # not-started, in-progress, completed completed_date: Optional[datetime] = None @dataclass class PostMortem: incident_id: str title: str date: datetime start_time: datetime end_time: datetime duration_minutes: int severity: IncidentSeverity affected_services: List[str] on_call_engineer: str post_mortem_lead: str status: str = "draft" # draft, review, final # Impact users_affected: Optional[int] = None revenue_impact: Optional[float] = None sla_breach: bool = False error_budget_consumption: Optional[float] = None # Content executive_summary: str = "" root_cause: str = "" contributing_factors: List[str] = field(default_factory=list) resolution: str = "" timeline: List[TimelineEvent] = field(default_factory=list) action_items: List[ActionItem] = field(default_factory=list) lessons_learned: Dict[str, List[str]] = field(default_factory=dict) # Metrics mttr_minutes: Optional[int] = None mttd_minutes: Optional[int] = None mtti_minutes: Optional[int] = None mttm_minutes: Optional[int] = None class PostMortemGenerator: def __init__(self): self.template_path = "templates/postmortem.md" def generate(self, postmortem: PostMortem) -> str: """Generate post-mortem document from data structure""" with open(self.template_path, 'r') as f: template = f.read() # Replace template variables content = template.replace("{{INCIDENT_ID}}", postmortem.incident_id) content = content.replace("{{TITLE}}", postmortem.title) content = content.replace("{{DATE}}", postmortem.date.strftime("%Y-%m-%d")) content = content.replace("{{DURATION}}", f"{postmortem.duration_minutes} minutes") content = content.replace("{{SEVERITY}}", postmortem.severity.value) content = content.replace("{{AFFECTED_SERVICES}}", ", ".join(postmortem.affected_services)) content = content.replace("{{EXECUTIVE_SUMMARY}}", postmortem.executive_summary) content = content.replace("{{ROOT_CAUSE}}", postmortem.root_cause) content = content.replace("{{RESOLUTION}}", postmortem.resolution) # Generate timeline table timeline_table = self._generate_timeline_table(postmortem.timeline) content = content.replace("{{TIMELINE}}", timeline_table) # Generate action items action_items = self._generate_action_items(postmortem.action_items) content = content.replace("{{ACTION_ITEMS}}", action_items) return content def _generate_timeline_table(self, events: List[TimelineEvent]) -> str: """Generate markdown table from timeline events""" rows = ["| Time | Event | Actor | Notes |", "|------|-------|-------|-------|"] for event in sorted(events, key=lambda e: e.time): rows.append(f"| {event.time.strftime('%H:%M %p')} | {event.event} | {event.actor} | {event.notes or ''} |") return "\n".join(rows) def _generate_action_items(self, items: List[ActionItem]) -> str: """Generate action items list""" sections = { "immediate": [], "short-term": [], "long-term": [] } for item in items: sections[item.priority].append(item) output = [] for priority, title in [("immediate", "Immediate (Next 24 hours)"), ("short-term", "Short-term (Next week)"), ("long-term", "Long-term (Next month)")]: if sections[priority]: output.append(f"### {title}") for item in sections[priority]: status_icon = "✅" if item.status == "completed" else "⏳" if item.status == "in-progress" else "📋" output.append(f"- {status_icon} **{item.description}** - Owner: {item.owner} - Due: {item.due_date.strftime('%Y-%m-%d')}") return "\n".join(output) def export_json(self, postmortem: PostMortem) -> str: """Export post-mortem as JSON""" data = { "incident_id": postmortem.incident_id, "title": postmortem.title, "date": postmortem.date.isoformat(), "severity": postmortem.severity.value, "affected_services": postmortem.affected_services, "timeline": [ { "time": event.time.isoformat(), "event": event.event, "actor": event.actor, "notes": event.notes } for event in postmortem.timeline ], "action_items": [ { "id": item.id, "description": item.description, "owner": item.owner, "due_date": item.due_date.isoformat(), "priority": item.priority, "status": item.status } for item in postmortem.action_items ] } return json.dumps(data, indent=2) # Example usage if __name__ == "__main__": generator = PostMortemGenerator() postmortem = PostMortem( incident_id="INC-20240115-1000", title="Database Connection Pool Exhaustion", date=datetime(2024, 1, 15), start_time=datetime(2024, 1, 15, 10, 0), end_time=datetime(2024, 1, 15, 10, 30), duration_minutes=30, severity=IncidentSeverity.SEV2, affected_services=["user-service", "payment-service"], on_call_engineer="engineer@example.com", post_mortem_lead="lead@example.com", executive_summary="Database connection pool was exhausted due to unclosed connections.", root_cause="Connection leaks in payment processing code", timeline=[ TimelineEvent(datetime(2024, 1, 15, 10, 0), "Alert fired", "Prometheus"), TimelineEvent(datetime(2024, 1, 15, 10, 2), "Engineer acknowledged", "Engineer A"), TimelineEvent(datetime(2024, 1, 15, 10, 30), "Incident resolved", "Engineer A"), ], action_items=[ ActionItem("AI-001", "Fix connection leaks", "dev@example.com", datetime(2024, 1, 16), "immediate"), ActionItem("AI-002", "Add connection pool monitoring", "sre@example.com", datetime(2024, 1, 22), "short-term"), ] ) markdown = generator.generate(postmortem) print(markdown) ``` ### 3. Blameless Post-Mortem Guidelines ```markdown # Blameless Post-Mortem Guidelines ## Principles ### 1. Focus on Systems, Not People - **DO:** "The deployment process didn't catch the configuration error" - **DON'T:** "John forgot to check the configuration" ### 2. Assume Good Intentions - Everyone was trying to do the right thing - Mistakes happen in complex systems - Focus on how the system allowed the mistake ### 3. Learn, Don't Blame - Goal is improvement, not punishment - Understand why decisions were made - Identify systemic issues ### 4. Be Specific and Factual - Use data and evidence - Avoid speculation - Document what actually happened ## Language Guidelines ### ✅ Good Language - "The system failed to..." - "The process didn't catch..." - "The monitoring didn't alert on..." - "We discovered that..." - "The design allowed..." ### ❌ Blaming Language - "X person made a mistake" - "X team should have..." - "X person forgot to..." - "X person didn't follow procedure" - "X person's fault" ## Meeting Structure 1. **Set Ground Rules** (5 min) - Remind everyone: blameless culture - Focus on learning - No finger-pointing 2. **Timeline Construction** (20 min) - Build timeline together - Everyone adds their perspective - No judgment, just facts 3. **Root Cause Discussion** (30 min) - Use "5 Whys" technique - Ask "why" until you reach systemic cause - Focus on processes, not people 4. **Action Items** (15 min) - Identify improvements - Assign owners - Set deadlines 5. **Wrap-up** (5 min) - Thank everyone - Remind about follow-up - Schedule review meeting ``` ### 4. Post-Mortem Checklist ```markdown # Post-Mortem Checklist ## Before the Meeting - [ ] Incident is fully resolved - [ ] All stakeholders notified - [ ] Meeting scheduled (within 24-48 hours) - [ ] Data collected: - [ ] Logs from all services - [ ] Metrics and dashboards - [ ] Timeline of events - [ ] Chat logs (Slack, etc.) - [ ] Deployment history - [ ] Draft template prepared - [ ] Meeting room/video link set up ## During the Meeting - [ ] Ground rules established (blameless) - [ ] Timeline constructed - [ ] Root cause identified - [ ] Contributing factors discussed - [ ] Action items created - [ ] Lessons learned captured - [ ] Next steps agreed upon ## After the Meeting - [ ] Post-mortem document finalized - [ ] Action items assigned and tracked - [ ] Post-mortem published (internal wiki/docs) - [ ] Team notified of publication - [ ] Follow-up review scheduled - [ ] Related teams notified if applicable ``` ### 5. Post-Mortem Metrics Dashboard ```yaml # grafana/postmortem-dashboard.json { "dashboard": { "title": "Post-Mortem Metrics", "panels": [ { "title": "MTTR by Severity", "targets": [ { "expr": "avg(postmortem_mttr_minutes) by (severity)", "legendFormat": "{{severity}}" } ], "type": "graph" }, { "title": "Action Items Completion Rate", "targets": [ { "expr": "sum(postmortem_action_items_completed) / sum(postmortem_action_items_total) * 100", "legendFormat": "Completion Rate" } ], "type": "stat" }, { "title": "Post-Mortems by Month", "targets": [ { "expr": "count(postmortem_total) by (month)", "legendFormat": "{{month}}" } ], "type": "graph" }, { "title": "Top Contributing Factors", "targets": [ { "expr": "topk(10, count(postmortem_contributing_factor) by (factor))", "legendFormat": "{{factor}}" } ], "type": "table" } ] } } ``` ## 🎯 Mejores Prácticas ### 1. Timing ✅ **DO:** - Schedule within 24-48 hours of incident - While memories are fresh - Before people move on to other work ❌ **DON'T:** - Wait more than a week - Schedule during high-stress periods - Rush the process ### 2. Participation ✅ **DO:** - Include all stakeholders - Include people who made decisions - Invite people who can provide context - Keep group size manageable (5-10 people) ❌ **DON'T:** - Exclude people who were involved - Make it a blame session - Include people who weren't involved ### 3. Documentation ✅ **DO:** - Write clearly and concisely - Use data and evidence - Include timeline with timestamps - Make action items specific and trackable - Publish where everyone can access ❌ **DON'T:** - Use vague language - Skip important details - Make action items vague - Hide post-mortems ### 4. Follow-up ✅ **DO:** - Track action items to completion - Review post-mortems periodically - Share learnings across teams - Update runbooks based on learnings - Measure improvement over time ❌ **DON'T:** - Create action items and forget them - Let post-mortems sit unread - Keep learnings siloed - Repeat the same mistakes ## 🚨 Troubleshooting ### Post-Mortem Not Happening 1. **Make it mandatory** for SEV1/SEV2 incidents 2. **Automate scheduling** after incident resolution 3. **Track compliance** as a metric 4. **Make it easy** with templates and tools ### Blame Culture 1. **Set ground rules** at start of every meeting 2. **Call out blaming language** when it happens 3. **Focus on systems** in all discussions 4. **Lead by example** from management ### Action Items Not Completed 1. **Assign clear owners** with deadlines 2. **Track in project management** tool 3. **Review in team meetings** 4. **Escalate if overdue** ### Low-Quality Post-Mortems 1. **Provide training** on post-mortem process 2. **Review and provide feedback** 3. **Share examples** of good post-mortems 4. **Make templates** more detailed ## 📚 Recursos Adicionales - [Google SRE - Postmortem Culture](https://sre.google/workbook/postmortem-culture/) - [Blameless Postmortems](https://www.blameless.com/blog/blameless-postmortem) - [Atlassian Post-Mortem Guide](https://www.atlassian.com/incident-management/handbook/postmortems) - [Post-Mortem Templates Collection](https://github.com/dastergon/awesome-sre#post-mortem) --- **Versión:** 1.0.0 **Última actualización:** Diciembre 2025 **Total líneas:** 800+