Incident Response Runbook

Incident Response Runbook

Execute structured incident response for: {{ incident_title }} Severity: {{ severity }} | Service: {{ affected_service }}

Severity Decision Matrix

Severity-specific playbooks with escalation matrices

Workflow

Phase 1 — DETECT & DECLARE (0-5 min)

1. Confirm incident is real — not a monitoring false positive

   • Check dashboard for corroborating signals
   • Verify user impact exists (not just synthetic test failure)

2. Declare the incident

   • Assign Incident Commander (IC) — single decision-maker
   • Open incident channel: {{ incident_channel }}
   • Post initial broadcast:

   INCIDENT DECLARED — {{ severity }}
   Title: {{ incident_title }}
   Affected: {{ affected_service }}
   IC: [your name]
   Bridge: {{ incident_channel }}
   Status: INVESTIGATING
   

3. Page on-call if not already paged

   • Escalate to secondary if primary does not ACK within 5 min (SEV1) / 15 min (SEV2)

Phase 2 — TRIAGE (5-15 min)

Answer ALL five triage questions — do not skip any:

Phase 3 — INVESTIGATE (15-60 min)

Systematic investigation order:

1. Dashboards — error rate, latency, throughput
2. Logs — first error occurrence, error patterns
3. Recent changes — deployment history, config changes
4. Dependencies — database, cache, external API status
5. Capacity — CPU, memory, disk, connection pool

Investigation log (IC maintains):

[HH:MM] SIGNAL: [what was observed]
[HH:MM] ACTION: [what was tried]
[HH:MM] RESULT: [what happened]
[HH:MM] HYPOTHESIS: [current best guess]
[HH:MM] REJECTED: [why hypothesis was wrong] ← capture failed hypotheses too

Status updates (every 15 min to {{ incident_channel }}):

STATUS UPDATE — {{ severity }}
Elapsed: [X] minutes
Status: INVESTIGATING / MITIGATING / MONITORING
Hypothesis: [one sentence]
Next action: [what is being tried]
ETA: [estimate or "unknown"]

Phase 4 — MITIGATE (variable)

Apply mitigation in risk-priority order (lowest risk first):

Document the decision:

MITIGATION: [action]
Reason: [why chosen over alternatives]
Risk: [what could go wrong]
Rollback plan: [how to undo if it makes things worse]
Approved by IC at [HH:MM]

Phase 5 — RESOLVE

Incident is resolved when ALL conditions met:

• Error rate back to baseline (<1% or pre-incident level)
• Latency back to baseline
• All services reporting healthy
• No new customer complaints

Resolution announcement:

INCIDENT RESOLVED — {{ severity }}
Title: {{ incident_title }}
Duration: [X hours Y minutes]
Root Cause: [one sentence]
Resolution: [what fixed it]
Post-mortem: Scheduled for [date]

Post-resolution checklist:

• Update status page
• Notify customer success team
• Create post-mortem ticket (Jira/Linear)
• Schedule post-mortem meeting (within 48h for SEV1/SEV2)
• Close PagerDuty incident

Phase 6 — POST-MORTEM

Create within 48h for SEV1/SEV2. Structure:

INCIDENT POST-MORTEM
Title: {{ incident_title }}
Date: [date] | Duration: [total] | Severity: {{ severity }}

IMPACT: [users affected, features degraded, revenue impact]
TIMELINE: [chronological key events]
ROOT CAUSE: [technical cause — be specific, avoid blame]
CONTRIBUTING FACTORS: [systemic issues]
WHAT WENT WELL: [things that worked]
WHAT TO IMPROVE: [things that slowed response]

ACTION ITEMS:
| Action | Owner | Due Date | Priority |
|--------|-------|----------|----------|

Counter-Rationalizations

Output Format

Produce a real-time incident log:

1. Header — title, severity, IC, start time
2. Triage answers — all 5 questions answered
3. Investigation findings — signals, hypotheses, evidence
4. Mitigation decision — action, reasoning, rollback plan
5. Resolution — what fixed it, duration, next steps
6. Post-mortem draft — on request

References

• Severity Playbooks — Escalation Matrices & Investigation Runbooks