Post Mortem

📋 Skill: Post-Mortem Process

📋 Metadata

🔑 Keywords para Invocación

• post-mortem
• postmortem
• incident-review
• blameless
• lessons-learned
• root-cause-analysis
• incident-analysis
• @skill:post-mortem

Ejemplos de Prompts

Crea un post-mortem para el incidente de ayer

Implementa proceso de post-mortem blameless

Genera template de post-mortem con timeline y action items

@skill:post-mortem - Proceso completo de análisis de incidentes

📖 Descripción

Un post-mortem es un análisis estructurado y blameless de un incidente que busca entender qué pasó, por qué pasó, y cómo prevenir que vuelva a ocurrir. Este skill cubre la metodología completa de post-mortems, templates, mejores prácticas, y herramientas para facilitar el proceso.

✅ Cuándo Usar Este Skill

• Después de incidentes críticos (SEV1, SEV2)
• Cuando se consume error budget significativo
• Incidentes que afectan múltiples usuarios
• Incidentes que requieren escalación
• Para cumplir con compliance requirements
• Cuando hay patrones recurrentes de incidentes

❌ Cuándo NO Usar Este Skill

• Incidentes menores sin impacto (SEV4)
• Problemas resueltos en < 5 minutos sin impacto
• Incidentes de mantenimiento planificados
• Cuando no hay tiempo para análisis adecuado

🏗️ Proceso de Post-Mortem

┌─────────────────┐
│ Incident Occurs │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Incident Resolved│
└────────┬────────┘
         │
         │ (Within 24-48 hours)
         ▼
┌─────────────────┐
│ Schedule Meeting │
│ (All Stakeholders)│
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Collect Data    │
│ - Logs          │
│ - Metrics       │
│ - Timeline      │
│ - Testimonies   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Write Draft     │
│ Post-Mortem     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Review Meeting  │
│ (Blameless)     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Finalize &      │
│ Publish         │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Track Action    │
│ Items           │
└─────────────────┘

💻 Implementación

1. Post-Mortem Template

# Post-Mortem: [Incident Title]

**Incident ID:** [INC-YYYYMMDD-HHMMSS]
**Date:** [Date]
**Duration:** [Start Time] - [End Time] (X minutes/hours)
**Severity:** [SEV1/SEV2/SEV3]
**Affected Services:** [Service names]
**On-Call Engineer:** [Name]
**Post-Mortem Lead:** [Name]
**Status:** [Draft/Review/Final]

---

## Executive Summary

[2-3 sentence summary for leadership. What happened, impact, and resolution.]

**TL;DR:** [One sentence summary]

---

## Impact

### User Impact
- **Users Affected:** [Number or percentage]
- **Geographic Impact:** [Regions affected]
- **Feature Impact:** [Which features were unavailable]

### Business Impact
- **Revenue Impact:** [If applicable, estimated $]
- **SLA Breach:** [Yes/No - which SLA]
- **Error Budget Consumption:** [X% of monthly budget]
- **Customer Complaints:** [Number]

### Technical Impact
- **Services Down:** [List]
- **Data Loss:** [Yes/No - details]
- **Performance Degradation:** [Metrics]

---

## Timeline

| Time (UTC) | Event | Actor | Notes |
|------------|-------|-------|-------|
| 10:00 AM | Alert fired: HighErrorRate | Prometheus | Error rate > 5% |
| 10:02 AM | On-call engineer acknowledged | Engineer A | PagerDuty notification |
| 10:05 AM | Initial investigation started | Engineer A | Checking logs |
| 10:10 AM | Identified pattern in logs | Engineer A | Database connection errors |
| 10:15 AM | Root cause identified | Engineer A | Connection pool exhaustion |
| 10:20 AM | Mitigation deployed | Engineer B | Increased pool size |
| 10:25 AM | Service restored | System | Metrics normalized |
| 10:30 AM | Incident resolved | Engineer A | All systems healthy |

**Key Milestones:**
- **Detection Time:** 10:00 AM
- **Response Time:** 2 minutes
- **Time to Identify:** 15 minutes
- **Time to Mitigate:** 20 minutes
- **Time to Resolve:** 30 minutes
- **Total Downtime:** 30 minutes

---

## Root Cause Analysis

### Primary Root Cause

[Detailed explanation of what caused the incident. Use the "5 Whys" technique if applicable.]

**What Happened:**
[Description of the immediate cause]

**Why It Happened:**
[Underlying technical reason]

**Why That Happened:**
[Deeper systemic reason]

### Contributing Factors

1. **[Factor 1]**
   - **Impact:** [How it contributed]
   - **Why it existed:** [Systemic reason]

2. **[Factor 2]**
   - **Impact:** [How it contributed]
   - **Why it existed:** [Systemic reason]

3. **[Factor 3]**
   - **Impact:** [How it contributed]
   - **Why it existed:** [Systemic reason]

### Detection Gaps

- [What we should have detected earlier]
- [Why we didn't detect it]
- [What monitoring/metrics were missing]

---

## Resolution

### Immediate Actions Taken

1. **[Action 1]**
   - **Who:** [Name]
   - **When:** [Time]
   - **Result:** [Outcome]

2. **[Action 2]**
   - **Who:** [Name]
   - **When:** [Time]
   - **Result:** [Outcome]

### Long-term Fix

[What was done to permanently resolve the issue]

---

## Action Items

### Immediate (Next 24 hours)
- [ ] **Item 1** - Owner: [Name] - Due: [Date]
  - Description: [Details]
- [ ] **Item 2** - Owner: [Name] - Due: [Date]
  - Description: [Details]

### Short-term (Next week)
- [ ] **Item 1** - Owner: [Name] - Due: [Date]
  - Description: [Details]
- [ ] **Item 2** - Owner: [Name] - Due: [Date]
  - Description: [Details]

### Long-term (Next month)
- [ ] **Item 1** - Owner: [Name] - Due: [Date]
  - Description: [Details]
- [ ] **Item 2** - Owner: [Name] - Due: [Date]
  - Description: [Details]

**Action Item Tracking:**
- Total Items: [X]
- Completed: [Y]
- In Progress: [Z]
- Not Started: [W]

---

## Lessons Learned

### What Went Well ✅

1. **[Positive aspect 1]**
   - **Why it helped:** [Explanation]
   - **How to replicate:** [Actionable steps]

2. **[Positive aspect 2]**
   - **Why it helped:** [Explanation]
   - **How to replicate:** [Actionable steps]

### What Could Be Improved 🔄

1. **[Improvement 1]**
   - **Current state:** [What happened]
   - **Desired state:** [What should happen]
   - **How to improve:** [Actionable steps]

2. **[Improvement 2]**
   - **Current state:** [What happened]
   - **Desired state:** [What should happen]
   - **How to improve:** [Actionable steps]

### Surprises 🤔

- [Unexpected behavior or findings]
- [Things we learned about the system]

---

## Prevention

### How to Prevent This Incident

1. **[Prevention measure 1]**
   - **Type:** [Monitoring/Process/Code/Infrastructure]
   - **Priority:** [High/Medium/Low]
   - **Effort:** [Small/Medium/Large]

2. **[Prevention measure 2]**
   - **Type:** [Monitoring/Process/Code/Infrastructure]
   - **Priority:** [High/Medium/Low]
   - **Effort:** [Small/Medium/Large]

### Similar Incidents

- [Link to related post-mortems]
- [Patterns identified]

---

## Follow-up

### Post-Mortem Review

- **Scheduled Date:** [Date]
- **Participants:** [Names]
- **Status:** [Scheduled/Completed]

### Action Item Review

- **Next Review Date:** [Date]
- **Owner:** [Name]
- **Last Updated:** [Date]

### Communication

- **Internal:** [Who was notified]
- **External:** [Customer communication if applicable]
- **Status Updates:** [Where updates were posted]

---

## Metrics

### Incident Metrics

- **MTTR (Mean Time To Resolution):** [X minutes]
- **MTTD (Mean Time To Detection):** [X minutes]
- **MTTI (Mean Time To Identify):** [X minutes]
- **MTTM (Mean Time To Mitigate):** [X minutes]

### Historical Comparison

- **Similar incidents in last 6 months:** [Number]
- **Trend:** [Improving/Stable/Worsening]

---

## Appendix

### Logs & Evidence

- [Links to relevant logs]
- [Screenshots of dashboards]
- [Metrics graphs]

### Related Documentation

- [Runbooks used]
- [Related post-mortems]
- [Design documents]

### Team Feedback

- [Quotes from team members]
- [Additional context]

---

**Document Version:** 1.0
**Last Updated:** [Date]
**Next Review:** [Date]

2. Post-Mortem Automation Script

# postmortem/generator.py
from dataclasses import dataclass, field
from datetime import datetime
from typing import List, Optional, Dict
from enum import Enum
import json

class IncidentSeverity(Enum):
    SEV1 = "critical"  # Service down
    SEV2 = "high"      # Major degradation
    SEV3 = "medium"    # Minor issues
    SEV4 = "low"       # Informational

@dataclass
class TimelineEvent:
    time: datetime
    event: str
    actor: str
    notes: Optional[str] = None

@dataclass
class ActionItem:
    id: str
    description: str
    owner: str
    due_date: datetime
    priority: str  # immediate, short-term, long-term
    status: str = "not-started"  # not-started, in-progress, completed
    completed_date: Optional[datetime] = None

@dataclass
class PostMortem:
    incident_id: str
    title: str
    date: datetime
    start_time: datetime
    end_time: datetime
    duration_minutes: int
    severity: IncidentSeverity
    affected_services: List[str]
    on_call_engineer: str
    post_mortem_lead: str
    status: str = "draft"  # draft, review, final

    # Impact
    users_affected: Optional[int] = None
    revenue_impact: Optional[float] = None
    sla_breach: bool = False
    error_budget_consumption: Optional[float] = None

    # Content
    executive_summary: str = ""
    root_cause: str = ""
    contributing_factors: List[str] = field(default_factory=list)
    resolution: str = ""
    timeline: List[TimelineEvent] = field(default_factory=list)
    action_items: List[ActionItem] = field(default_factory=list)
    lessons_learned: Dict[str, List[str]] = field(default_factory=dict)

    # Metrics
    mttr_minutes: Optional[int] = None
    mttd_minutes: Optional[int] = None
    mtti_minutes: Optional[int] = None
    mttm_minutes: Optional[int] = None

class PostMortemGenerator:
    def __init__(self):
        self.template_path = "templates/postmortem.md"

    def generate(self, postmortem: PostMortem) -> str:
        """Generate post-mortem document from data structure"""
        with open(self.template_path, 'r') as f:
            template = f.read()

        # Replace template variables
        content = template.replace("{{INCIDENT_ID}}", postmortem.incident_id)
        content = content.replace("{{TITLE}}", postmortem.title)
        content = content.replace("{{DATE}}", postmortem.date.strftime("%Y-%m-%d"))
        content = content.replace("{{DURATION}}", f"{postmortem.duration_minutes} minutes")
        content = content.replace("{{SEVERITY}}", postmortem.severity.value)
        content = content.replace("{{AFFECTED_SERVICES}}", ", ".join(postmortem.affected_services))
        content = content.replace("{{EXECUTIVE_SUMMARY}}", postmortem.executive_summary)
        content = content.replace("{{ROOT_CAUSE}}", postmortem.root_cause)
        content = content.replace("{{RESOLUTION}}", postmortem.resolution)

        # Generate timeline table
        timeline_table = self._generate_timeline_table(postmortem.timeline)
        content = content.replace("{{TIMELINE}}", timeline_table)

        # Generate action items
        action_items = self._generate_action_items(postmortem.action_items)
        content = content.replace("{{ACTION_ITEMS}}", action_items)

        return content

    def _generate_timeline_table(self, events: List[TimelineEvent]) -> str:
        """Generate markdown table from timeline events"""
        rows = ["| Time | Event | Actor | Notes |", "|------|-------|-------|-------|"]
        for event in sorted(events, key=lambda e: e.time):
            rows.append(f"| {event.time.strftime('%H:%M %p')} | {event.event} | {event.actor} | {event.notes or ''} |")
        return "\n".join(rows)

    def _generate_action_items(self, items: List[ActionItem]) -> str:
        """Generate action items list"""
        sections = {
            "immediate": [],
            "short-term": [],
            "long-term": []
        }

        for item in items:
            sections[item.priority].append(item)

        output = []
        for priority, title in [("immediate", "Immediate (Next 24 hours)"),
                                ("short-term", "Short-term (Next week)"),
                                ("long-term", "Long-term (Next month)")]:
            if sections[priority]:
                output.append(f"### {title}")
                for item in sections[priority]:
                    status_icon = "✅" if item.status == "completed" else "⏳" if item.status == "in-progress" else "📋"
                    output.append(f"- {status_icon} **{item.description}** - Owner: {item.owner} - Due: {item.due_date.strftime('%Y-%m-%d')}")

        return "\n".join(output)

    def export_json(self, postmortem: PostMortem) -> str:
        """Export post-mortem as JSON"""
        data = {
            "incident_id": postmortem.incident_id,
            "title": postmortem.title,
            "date": postmortem.date.isoformat(),
            "severity": postmortem.severity.value,
            "affected_services": postmortem.affected_services,
            "timeline": [
                {
                    "time": event.time.isoformat(),
                    "event": event.event,
                    "actor": event.actor,
                    "notes": event.notes
                }
                for event in postmortem.timeline
            ],
            "action_items": [
                {
                    "id": item.id,
                    "description": item.description,
                    "owner": item.owner,
                    "due_date": item.due_date.isoformat(),
                    "priority": item.priority,
                    "status": item.status
                }
                for item in postmortem.action_items
            ]
        }
        return json.dumps(data, indent=2)

# Example usage
if __name__ == "__main__":
    generator = PostMortemGenerator()

    postmortem = PostMortem(
        incident_id="INC-20240115-1000",
        title="Database Connection Pool Exhaustion",
        date=datetime(2024, 1, 15),
        start_time=datetime(2024, 1, 15, 10, 0),
        end_time=datetime(2024, 1, 15, 10, 30),
        duration_minutes=30,
        severity=IncidentSeverity.SEV2,
        affected_services=["user-service", "payment-service"],
       
        post_mortem_lead="lead@example.com",
        executive_summary="Database connection pool was exhausted due to unclosed connections.",
        root_cause="Connection leaks in payment processing code",
        timeline=[
            TimelineEvent(datetime(2024, 1, 15, 10, 0), "Alert fired", "Prometheus"),
            TimelineEvent(datetime(2024, 1, 15, 10, 2), "Engineer acknowledged", "Engineer A"),
            TimelineEvent(datetime(2024, 1, 15, 10, 30), "Incident resolved", "Engineer A"),
        ],
        action_items=[
            ActionItem("AI-001", "Fix connection leaks", "dev@example.com", datetime(2024, 1, 16), "immediate"),
            ActionItem("AI-002", "Add connection pool monitoring", "sre@example.com", datetime(2024, 1, 22), "short-term"),
        ]
    )

    markdown = generator.generate(postmortem)
    print(markdown)

3. Blameless Post-Mortem Guidelines

# Blameless Post-Mortem Guidelines

## Principles

### 1. Focus on Systems, Not People
- **DO:** "The deployment process didn't catch the configuration error"
- **DON'T:** "John forgot to check the configuration"

### 2. Assume Good Intentions
- Everyone was trying to do the right thing
- Mistakes happen in complex systems
- Focus on how the system allowed the mistake

### 3. Learn, Don't Blame
- Goal is improvement, not punishment
- Understand why decisions were made
- Identify systemic issues

### 4. Be Specific and Factual
- Use data and evidence
- Avoid speculation
- Document what actually happened

## Language Guidelines

### ✅ Good Language
- "The system failed to..."
- "The process didn't catch..."
- "The monitoring didn't alert on..."
- "We discovered that..."
- "The design allowed..."

### ❌ Blaming Language
- "X person made a mistake"
- "X team should have..."
- "X person forgot to..."
- "X person didn't follow procedure"
- "X person's fault"

## Meeting Structure

1. **Set Ground Rules** (5 min)
   - Remind everyone: blameless culture
   - Focus on learning
   - No finger-pointing

2. **Timeline Construction** (20 min)
   - Build timeline together
   - Everyone adds their perspective
   - No judgment, just facts

3. **Root Cause Discussion** (30 min)
   - Use "5 Whys" technique
   - Ask "why" until you reach systemic cause
   - Focus on processes, not people

4. **Action Items** (15 min)
   - Identify improvements
   - Assign owners
   - Set deadlines

5. **Wrap-up** (5 min)
   - Thank everyone
   - Remind about follow-up
   - Schedule review meeting

4. Post-Mortem Checklist

# Post-Mortem Checklist

## Before the Meeting
- [ ] Incident is fully resolved
- [ ] All stakeholders notified
- [ ] Meeting scheduled (within 24-48 hours)
- [ ] Data collected:
  - [ ] Logs from all services
  - [ ] Metrics and dashboards
  - [ ] Timeline of events
  - [ ] Chat logs (Slack, etc.)
  - [ ] Deployment history
- [ ] Draft template prepared
- [ ] Meeting room/video link set up

## During the Meeting
- [ ] Ground rules established (blameless)
- [ ] Timeline constructed
- [ ] Root cause identified
- [ ] Contributing factors discussed
- [ ] Action items created
- [ ] Lessons learned captured
- [ ] Next steps agreed upon

## After the Meeting
- [ ] Post-mortem document finalized
- [ ] Action items assigned and tracked
- [ ] Post-mortem published (internal wiki/docs)
- [ ] Team notified of publication
- [ ] Follow-up review scheduled
- [ ] Related teams notified if applicable

5. Post-Mortem Metrics Dashboard

# grafana/postmortem-dashboard.json
{
  "dashboard": {
    "title": "Post-Mortem Metrics",
    "panels": [
      {
        "title": "MTTR by Severity",
        "targets": [
          {
            "expr": "avg(postmortem_mttr_minutes) by (severity)",
            "legendFormat": "{{severity}}"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Action Items Completion Rate",
        "targets": [
          {
            "expr": "sum(postmortem_action_items_completed) / sum(postmortem_action_items_total) * 100",
            "legendFormat": "Completion Rate"
          }
        ],
        "type": "stat"
      },
      {
        "title": "Post-Mortems by Month",
        "targets": [
          {
            "expr": "count(postmortem_total) by (month)",
            "legendFormat": "{{month}}"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Top Contributing Factors",
        "targets": [
          {
            "expr": "topk(10, count(postmortem_contributing_factor) by (factor))",
            "legendFormat": "{{factor}}"
          }
        ],
        "type": "table"
      }
    ]
  }
}

🎯 Mejores Prácticas

1. Timing

✅ DO:

• Schedule within 24-48 hours of incident
• While memories are fresh
• Before people move on to other work

❌ DON'T:

• Wait more than a week
• Schedule during high-stress periods
• Rush the process

2. Participation

✅ DO:

• Include all stakeholders
• Include people who made decisions
• Invite people who can provide context
• Keep group size manageable (5-10 people)

❌ DON'T:

• Exclude people who were involved
• Make it a blame session
• Include people who weren't involved

3. Documentation

✅ DO:

• Write clearly and concisely
• Use data and evidence
• Include timeline with timestamps
• Make action items specific and trackable
• Publish where everyone can access

❌ DON'T:

• Use vague language
• Skip important details
• Make action items vague
• Hide post-mortems

4. Follow-up

✅ DO:

• Track action items to completion
• Review post-mortems periodically
• Share learnings across teams
• Update runbooks based on learnings
• Measure improvement over time

❌ DON'T:

• Create action items and forget them
• Let post-mortems sit unread
• Keep learnings siloed
• Repeat the same mistakes

🚨 Troubleshooting

Post-Mortem Not Happening

1. Make it mandatory for SEV1/SEV2 incidents
2. Automate scheduling after incident resolution
3. Track compliance as a metric
4. Make it easy with templates and tools

Blame Culture

1. Set ground rules at start of every meeting
2. Call out blaming language when it happens
3. Focus on systems in all discussions
4. Lead by example from management

Action Items Not Completed

1. Assign clear owners with deadlines
2. Track in project management tool
3. Review in team meetings
4. Escalate if overdue

Low-Quality Post-Mortems

1. Provide training on post-mortem process
2. Review and provide feedback
3. Share examples of good post-mortems
4. Make templates more detailed

📚 Recursos Adicionales

• Google SRE - Postmortem Culture
• Blameless Postmortems
• Atlassian Post-Mortem Guide
• Post-Mortem Templates Collection

Versión: 1.0.0 Última actualización: Diciembre 2025 Total líneas: 800+