Incident Response

Incident Response Rules

1. Severity Classification

Severity Levels

Severity Examples

Escalation Path

SEV1: On-call → Team Lead → Engineering Manager → VP Eng (within 30 min)
SEV2: On-call → Team Lead → Engineering Manager (within 1 hour)
SEV3: On-call → Team Lead (within 4 hours)
SEV4: Ticket in backlog

2. Incident Lifecycle

Phases

Detection → Triage → Mitigation → Resolution → Postmortem
    │          │          │            │            │
    ▼          ▼          ▼            ▼            ▼
  Alert     Assess     Stop the    Fix root     Learn and
  fires     impact     bleeding    cause        improve

Role Assignments

Role Assignment Rules

• IC is the first responder until explicitly handed off
• For SEV1/SEV2, assign all four roles within 15 minutes
• IC should NOT be debugging — they coordinate
• Rotate IC during long incidents (>4 hours)

3. Communication Protocol

Status Update Template

[Incident #1234] [SEV1] [UPDATE 3] [2024-01-15 14:30 UTC]

Status: MITIGATING
Impact: 100% of users unable to complete checkout
Root cause: Database connection pool exhausted due to connection leak
Mitigation: Rolling restart of affected services in progress
ETA: 15 minutes to full recovery
Next update: 14:45 UTC

Stakeholder Notification Matrix

Communication Channels

4. Triage Checklist

Initial Assessment (First 5 Minutes)

1. What is broken? — Identify the affected service/feature
2. Who is affected? — Estimate user impact (all users, specific region, specific plan)
3. When did it start? — Check monitoring for the onset time
4. What changed? — Review recent deployments, config changes, infrastructure events
5. Is it getting worse? — Check if error rate is increasing or stable

Blast Radius Estimation

Quick Diagnostic Commands

# Check recent deployments
kubectl rollout history deployment/<app> -n <namespace>

# Check pod health
kubectl get pods -n <namespace> -o wide | grep -v Running

# Check recent logs for errors
kubectl logs -n <namespace> -l app=<app> --since=10m | grep -i error | tail -20

# Check resource pressure
kubectl top pods -n <namespace>

# Check events
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

5. Mitigation Strategies

Decision Tree

Is the issue caused by a recent deployment?
├── Yes → Rollback deployment
│         └── Still broken? → Check config changes
└── No
    ├── Is traffic volume abnormal?
    │   ├── Yes → Scale up / enable rate limiting
    │   └── No → Continue investigation
    ├── Is a dependency down?
    │   ├── Yes → Enable circuit breaker / failover
    │   └── No → Continue investigation
    └── Is data corrupted?
        ├── Yes → Stop writes, assess damage, plan recovery
        └── No → Deep investigation needed

Mitigation Techniques

Mitigation Rules

• Prefer reversible actions (rollback, feature flag) over forward fixes during active incidents
• Communicate mitigation actions before executing them
• Document every action taken with timestamps
• If mitigation does not work within 15 minutes, escalate

6. Runbook Authoring

Runbook Template

# Runbook: [Service] — [Scenario]

## Overview
- **Service**: [service name]
- **Alert**: [alert name that triggers this runbook]
- **Severity**: [typical severity]
- **Last updated**: [date]
- **Owner**: [team name]

## Prerequisites
- [ ] Access to [environment/tool]
- [ ] Permissions: [required roles]

## Diagnosis Steps
1. Check [metric/dashboard] at [URL]
2. Run: `[diagnostic command]`
3. Expected output: [description]
4. If [condition], proceed to Mitigation A
5. If [other condition], proceed to Mitigation B

## Mitigation A: [Name]
1. Run: `[command]`
2. Verify: `[verification command]`
3. Expected result: [description]

## Mitigation B: [Name]
1. Run: `[command]`
2. Verify: `[verification command]`

## Escalation
- If neither mitigation works within [time], escalate to [team/person]
- Contact: [escalation contact info]

## Post-Mitigation Verification
1. Confirm error rate returns to baseline
2. Confirm no data loss or corruption
3. Monitor for [time period] before declaring resolved

Runbook Rules

• Every SEV1-capable alert MUST have a linked runbook
• Runbooks must be tested quarterly (dry run)
• Include exact commands, not vague instructions
• Include verification steps after each action
• Link to relevant dashboards and documentation

7. Postmortem Process

Timing

Blameless Postmortem Template

# Postmortem: [Incident Title]

## Metadata
- **Incident #**: [ID]
- **Date**: [YYYY-MM-DD]
- **Duration**: [start — end, total duration]
- **Severity**: [SEV level]
- **Author**: [name]
- **Reviewers**: [names]

## Summary
[2-3 sentence summary of what happened and the impact]

## Impact
- **Users affected**: [number/percentage]
- **Revenue impact**: [if applicable]
- **Data impact**: [if applicable]
- **Duration**: [time from detection to resolution]

## Timeline (all times UTC)
| Time | Event |
| --- | --- |
| 09:00 | Deployment X rolled out |
| 09:15 | Alert fired: error rate >5% |
| 09:18 | IC assigned, triage started |
| 09:25 | Root cause identified: connection leak |
| 09:30 | Rollback initiated |
| 09:35 | Error rate returning to baseline |
| 09:45 | Incident resolved |

## Root Cause
[Detailed technical explanation of what went wrong and why]

## Detection
- How was the incident detected? [alert / user report / manual check]
- Could we have detected it sooner? [yes/no, explain]

## Resolution
[What actions were taken to resolve the incident]

## Lessons Learned
### What went well
- [item]

### What went poorly
- [item]

### Where we got lucky
- [item]

## Action Items
| Action | Owner | Priority | Deadline | Ticket |
| --- | --- | --- | --- | --- |
| Add connection pool monitoring | @engineer | P1 | 2024-01-22 | JIRA-123 |
| Update runbook with new scenario | @oncall | P2 | 2024-01-29 | JIRA-124 |

Root Cause Analysis Techniques

Postmortem Rules

• MUST be blameless — focus on systems, not individuals
• All action items must have owners and deadlines
• Review action items in the next sprint/iteration
• Share postmortem with the broader engineering team
• Track recurring root causes to identify systemic issues

8. On-Call Best Practices

Rotation Design

Alert Fatigue Reduction

• Review alert signal-to-noise ratio monthly
• Suppress alerts during planned maintenance windows
• Group related alerts to reduce notification volume
• Set appropriate thresholds — avoid alerting on transient spikes
• Every alert must be actionable — if no action needed, remove it
• Target: <5 pages per on-call shift (excluding false positives)

Escalation Policy

Level 1: Primary on-call (immediate)
Level 2: Secondary on-call (after 15 min no-ack)
Level 3: Team lead (after 30 min no-ack)
Level 4: Engineering manager (after 45 min no-ack)

9. Metrics and KPIs

Key Incident Metrics

SLO / SLI / SLA Relationships

• SLO should be stricter than SLA — internal buffer
• Error budget = 1 - SLO (e.g., 0.1% = 43.2 min/month downtime budget)
• When error budget is exhausted, freeze non-critical deployments

10. Anti-Patterns

• Blaming individuals in postmortems — destroys psychological safety
• No runbooks for critical alerts — responders waste time investigating from scratch
• Skipping postmortems for SEV1/SEV2 — same incidents will recur
• IC also debugging — coordination suffers, nobody has the full picture
• Alerting on symptoms without context — responders cannot triage quickly
• Not tracking action items from postmortems — lessons are not learned
• Over-escalating every issue to SEV1 — severity inflation erodes urgency
• No handoff documentation between on-call shifts — context is lost
• Deploying during active incidents — adds more variables to troubleshoot
• No regular gameday drills — team discovers process gaps during real incidents

Additional References

• For detailed severity classification criteria and escalation guidelines, see references/severity-classification.md
• For structured postmortem template and blameless review guidelines, see references/postmortem-template.md

Related Skills

• For secret leakage incidents and credential rotation procedures, see secrets-management skill
• For general debugging and diagnosis patterns, see troubleshooting skill
• For chaos engineering and gameday practices, see chaos-engineering skill