Incident Response

Incident Response

Incident response is the structured process of detecting, mitigating, communicating, and learning from production failures to minimise user impact and prevent recurrence.

When to Activate

• Triaging a production alert or on-call page
• Writing a postmortem after an incident
• Creating or updating a runbook for a service
• Defining severity levels and escalation paths for a team
• Setting up an on-call rotation
• Running an incident response drill or game day

Severity Classification

Escalation path:

• P0/P1: page on-call engineer → page on-call lead if not ack'd in 5 min → escalate to eng manager
• P2: page on-call engineer
• P3/P4: create ticket, no page

Incident Lifecycle

Detection → Triage → Mitigate → Communicate → Resolve → Review (Postmortem)

First 5 Minutes — Triage Checklist

• Acknowledge the alert and claim the incident in your incident tool (PagerDuty / Opsgenie)
• Identify: what is broken, who is affected, since when?
• Check the deployment timeline: was anything deployed in the last 2 hours?
• Check the dashboards: error rate, latency, saturation — which service is the origin?
• Open an incident channel: #inc-YYYY-MM-DD-short-description
• Post initial acknowledgement message (see template below)
• Assign roles: Incident Commander (IC), Communicator, Subject Matter Expert (SME)

Communication Templates

Initial Acknowledgement

🔴 [P0/P1 INCIDENT] Payment service degradation

Status: Investigating
Impact: ~30% of payment requests failing with 500 errors since 14:23 UTC
Affected: All users attempting checkout

IC: @alice
SME: @bob
Next update: 14:45 UTC

Tracking: https://incident.example.com/inc-2024-0042

Status Update (every 15–30 min for P0/P1)

🟡 [P1 UPDATE] Payment service — 14:45 UTC

Status: Mitigating
Root cause identified: Connection pool exhaustion after deploy at 14:15
Action taken: Rolled back to v2.3.1, monitoring error rate
Current error rate: 2% (down from 30%)

Next update: 15:00 UTC

Resolution

✅ [P1 RESOLVED] Payment service — 15:02 UTC

Status: Resolved
Duration: 39 minutes (14:23 – 15:02 UTC)
Root cause: Deploy v2.4.0 introduced a connection leak; pool exhausted under load
Resolution: Rolled back to v2.3.1; error rate returned to baseline at 15:00

Users impacted: ~15,000 failed checkout attempts
Follow-up: Postmortem scheduled for 2024-01-16 15:00 UTC
Incident report: https://incident.example.com/inc-2024-0042

Mitigation Decision Tree

Error rate > SLO threshold?
├── Yes
│   ├── Was something deployed in the last 2 hours?
│   │   ├── Yes → ROLLBACK first, investigate after
│   │   └── No  → Check: DB, cache, upstream dependency, config change
│   ├── Can we isolate the impact with a feature flag kill?
│   │   └── Yes → Kill the flag immediately
│   └── Is this a traffic spike?
│       └── Yes → Scale up horizontally, enable circuit breaker
└── No — latency degraded only?
    ├── Check DB: slow queries, lock contention, pool saturation
    ├── Check cache hit rate: has cache been evicted?
    └── Check upstream service latency

When NOT to roll back immediately:

• The new version fixes a critical security issue (rolling back re-introduces the vulnerability)
• Rollback would itself cause data migration issues
• The issue is cosmetic (P3/P4) and the fix is already in progress

Runbook Structure

Runbooks must be written for the 3am engineer who has never seen this service.

# Runbook: [Service Name] — [Alert Name]

## Service Overview
[2–3 sentences: what does this service do, what does it depend on?]

## Alert: [Alert Name]
**Trigger condition:** [e.g., error rate > 1% for 5 minutes]
**Severity:** P1
**Dashboard:** [link]
**Logs:** [link to log query]

## Diagnostic Steps
1. Check the error rate panel on the [service dashboard](link)
   - Expected: < 0.1%
   - If > 1%: proceed to step 2
2. Check recent deployments:
   ```bash
   kubectl rollout history deployment/payment-service -n production

3. Check DB connection pool:

   kubectl exec -it $(kubectl get pod -l app=payment-service -o name | head -1) \
     -- curl -s localhost:8080/metrics | grep db_pool
   

   • If db_pool_wait_duration_seconds > 1s: pool is exhausted, proceed to step 4
4. Check for slow queries:

   SELECT query, mean_exec_time, calls
   FROM pg_stat_statements
   ORDER BY mean_exec_time DESC
   LIMIT 10;
   

Mitigation Steps

• If recent deployment: kubectl rollout undo deployment/payment-service -n production
• If DB pool exhausted: Scale up replicas: kubectl scale deployment/payment-service --replicas=6
• If upstream dependency: Enable circuit breaker feature flag: [link to flag]

Escalation

• If not resolved in 30 minutes: page @payment-team-lead
• DB issues: page @dba-on-call
• Infrastructure: page @infra-on-call

Related Runbooks

• Database connection issues
• High memory usage

**Runbook quality checks:**
- Every step has an expected output — the engineer knows what "normal" looks like
- Commands are copy-paste ready (no placeholders that need substitution)
- Decision points have explicit branches ("if X, do Y; if Z, do W")
- Links to dashboards, log queries, and escalation contacts are current

## Blameless Postmortem

Write the postmortem within 48 hours while details are fresh. **Blameless = focus on systems and processes, not individuals.**

```markdown
# Postmortem: [Service] [Brief Description] — [Date]

## Summary
[2–3 sentences: what happened, impact, how it was resolved]

**Impact:** [number of users affected, % error rate, duration]
**Detection time:** [how long from start to detection]
**Resolution time:** [how long from detection to resolution]

## Timeline (UTC)
| Time  | Event |
|-------|-------|
| 14:15 | Deploy v2.4.0 rolled out to 100% |
| 14:23 | Alert fired: error rate > 1% |
| 14:28 | On-call acknowledged, started investigation |
| 14:38 | Root cause identified: connection pool exhausted |
| 14:45 | Rollback initiated |
| 15:00 | Error rate returned to baseline |
| 15:02 | Incident declared resolved |

## Root Cause Analysis (5 Whys)
1. **Why** did payment requests fail?
   → DB connection pool was exhausted
2. **Why** was the pool exhausted?
   → v2.4.0 introduced a connection leak in the retry handler
3. **Why** did the retry handler leak connections?
   → The `defer conn.Close()` was placed inside the retry loop, closing on each attempt but not releasing the acquired connection back to the pool
4. **Why** wasn't this caught in testing?
   → Integration tests used a single-connection test DB; pool exhaustion only manifests at scale
5. **Why** wasn't this caught by the integration test DB pool?
   → Test pool size was set to 100 (no practical limit); prod pool size is 20

## Contributing Factors
- No load test run before this deploy
- No DB pool exhaustion alert existed
- Code review missed the subtle connection lifecycle issue

## What Went Well
- Alert fired within 8 minutes of degradation starting
- On-call was paged and acknowledged quickly
- Rollback decision was made in < 10 minutes

## Action Items

| Action | Owner | Due | Category |
|--------|-------|-----|----------|
| Add DB pool wait time alert (threshold: > 1s for 5 min) | @alice | 2024-01-19 | Detection |
| Add integration test that simulates pool exhaustion under concurrent load | @bob | 2024-01-26 | Prevention |
| Add `db_pool_size` check to pre-deploy checklist | @alice | 2024-01-19 | Prevention |
| Run k6 load test before all deploys touching DB connection code | @bob | 2024-01-26 | Prevention |

Action Item Categories

• Prevention: stops this class of failure from happening
• Detection: reduces time-to-detection (MTTD)
• Response: reduces time-to-resolution (MTTR)

Metrics to Track

Review these monthly per service. Rising MTTR = runbooks need updating. Repeat incidents = action items not implemented.

See also: observability, deployment-strategies

Red Flags

• Postmortem that names individuals as root cause — "Alice deployed bad code" stops at the human rather than the system that allowed the bad code to reach production; blameless postmortems ask why the system made it possible
• Action items with no owner or no due date — "Improve monitoring" as an action item is never done; every item must have a named owner and a specific due date to be tracked and closed
• Runbook that assumes the on-call engineer knows the service — runbooks must include what "normal" looks like and copy-paste commands; a 3am engineer touching an unfamiliar service cannot safely improvise
• Rolling back immediately without checking if the rollback itself causes data loss — rolling back a deploy that ran a destructive migration may orphan or corrupt rows that were written against the new schema
• Posting a P0 incident only in an engineering Slack channel — stakeholders (product, support, leadership) need timely updates via their own channels; the Communicator role exists specifically to bridge this gap
• Severity P0 declared for every outage regardless of blast radius — "P0" becomes meaningless if used for single-user bugs; a calibrated P0 ensures the right resources are mobilized and avoids on-call fatigue
• MTTD and MTTR tracked per-incident but never aggregated — individual numbers without a monthly trend hide whether the team is improving; review rolling averages per service each month
• Closing an incident before a postmortem is scheduled — if the postmortem is not scheduled at resolution time it rarely happens; require a postmortem date as a condition of closing any P0 or P1

Checklist

• Incident acknowledged within SLA (P0: 5 min, P1: 15 min)
• Incident channel opened and IC/SME roles assigned
• Initial acknowledgement posted to stakeholder channel
• Status updates sent on cadence (every 15 min for P0, 30 min for P1)
• Resolution announcement sent with impact summary
• Postmortem written within 48 hours of resolution
• 5 Whys root cause analysis complete (not just "human error")
• Action items are SMART: owner, due date, and category (prevention/detection/response)
• Runbook updated based on lessons learned
• MTTD, MTTA, MTTR recorded for this incident