Incident Response

Purpose

Handle production incidents with speed, clarity, and accountability. This skill provides frameworks for on-call response, structured communication, rollback execution, and learning from failures.

Severity Classification

• SEV-1 (Critical): Complete service outage, data loss, or security breach. All hands on deck. Resolve in minutes.
• SEV-2 (Major): Significant degradation affecting most users. Dedicated incident commander. Target resolution under 1 hour.
• SEV-3 (Minor): Partial degradation, workaround available. On-call engineer handles. Target resolution under 4 hours.
• SEV-4 (Low): Cosmetic issues, minor bugs, non-user-facing failures. Handle during business hours.
• Classify based on user impact, not technical complexity. A simple bug affecting all users is higher severity than a complex bug affecting one.
• Escalate severity upward when impact grows. Never downgrade severity during an active incident.

On-Call Runbooks

BAD - Vague runbook:

## Database Issues
If the database is slow, check the connections and maybe restart it.
Contact the DBA if it doesn't work.

GOOD - Actionable runbook with copy-paste commands:

## Runbook: Database Connection Pool Exhausted

### Symptoms
- API returns 503 errors
- Grafana alert: `db_active_connections > 95% of pool_size`
- Logs: "connection pool exhausted, 0 idle connections"

### Diagnose (< 1 minute)
    kubectl exec -it deploy/api -- curl localhost:8080/debug/db
    # Shows: active=100, idle=0, max=100, waiting=47

    psql -h $DB_HOST -c "SELECT count(*) FROM pg_stat_activity WHERE state='active';"

### Fix (choose one)
1. **Kill idle transactions** (safest):
       psql -h $DB_HOST -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle in transaction' AND query_start < now() - interval '5 min';"

2. **Increase pool size** (temporary):
       kubectl set env deploy/api DB_POOL_SIZE=200
       kubectl rollout restart deploy/api

3. **Rollback last deploy** (if caused by new code):
       kubectl rollout undo deploy/api

### Do NOT
- Do NOT restart the database. Active transactions will be lost.
- Do NOT increase pool size above 200 without DBA approval.

### Escalation
- Slack: #team-platform | PagerDuty: Platform On-Call

Write runbooks for every known failure mode. Structure: symptom, diagnostic commands, fix steps, escalation contacts. Store where on-call engineers find them in 30 seconds. Link directly from monitoring alerts. Review quarterly.

Incident Command Structure

• Assign an Incident Commander (IC) immediately. The IC coordinates, does not debug.
• The IC's responsibilities: delegate tasks, track progress, communicate externally, decide on escalation.
• Assign a Communications Lead for SEV-1 and SEV-2 incidents. They update the status page and notify stakeholders.
• Engineers working the incident report findings to the IC, not to each other. Single channel of communication.
• Use a dedicated incident channel (Slack, Teams). Name it #inc-YYYY-MM-DD-description.
• The IC calls the all-clear. No one else declares the incident resolved.

Communication Protocols

• First update within 5 minutes of detection for SEV-1, 15 minutes for SEV-2.
• Update every 30 minutes during an active incident, even if there is no progress. Silence causes panic.
• Use structured updates: current status, what we know, what we are doing, next update time.
• Internal updates go to the engineering team and leadership. External updates go to the status page.
• Avoid blame language during the incident. Focus on symptoms and actions.
• After resolution, send a final summary: root cause, fix applied, user impact duration, follow-up actions.

Status Page Management

• Maintain a public status page for external services. Update it before customers ask.
• Use clear, non-technical language: "Some users may experience slow page loads" over "Database replica lag exceeding threshold."
• Define status levels: Operational, Degraded Performance, Partial Outage, Major Outage.
• Pre-write templates for common scenarios. Fill in specifics during the incident.
• Include estimated resolution time when possible. Update it as the estimate changes.
• Post a follow-up after resolution confirming the issue is fully resolved.

Rollback Procedures

• Every deployment must have a documented rollback path before it ships.
• For application deployments: revert to the previous container image or artifact version.
• For database migrations: write and test the down migration before applying the up migration.
• For feature flags: disable the flag. This is the fastest rollback available.
• For infrastructure changes: use version-controlled IaC. Revert the commit and apply.
• Practice rollbacks in staging. A rollback you have never tested is a guess, not a plan.
• Set a time limit for debugging before rolling back. For SEV-1, roll back first, debug later.
• Preserve logs, metrics, and state before rolling back. You need evidence for the postmortem.

Blameless Postmortems

BAD - Blame-focused postmortem:

Root Cause: John deployed broken code on Friday at 5pm without testing.
Action Items: John needs to be more careful.

GOOD - Systems-focused postmortem:

## Postmortem: API Outage 2025-03-15

### Timeline (UTC)
- 16:42 - Deploy v2.3.1 to production (automated via merge to main)
- 16:44 - Error rate spikes from 0.1% to 34%
- 16:47 - PagerDuty alert fires, on-call acknowledges
- 16:52 - IC declared, #inc-2025-03-15-api-errors created
- 16:58 - Root cause identified: missing DB migration
- 17:01 - Rollback initiated (kubectl rollout undo)
- 17:03 - Error rate returns to 0.1%, all-clear declared

### Impact
- Duration: 21 minutes
- Users affected: ~12,000 (API returned 500 on /orders endpoint)
- Revenue impact: ~$3,200 in failed checkouts

### Root Cause
Deploy v2.3.1 added a query on `orders.status_v2` column. The migration
to add this column was not included in the deploy pipeline.

### Contributing Factors
- CI pipeline does not run pending migrations before integration tests
- No pre-deploy check that verifies schema compatibility
- Deploy happened automatically on merge, no manual gate

### Action Items
| Action | Owner | Due |
|--------|-------|-----|
| Add migration check to CI pipeline | @platform | 2025-03-22 |
| Add schema compatibility pre-deploy hook | @platform | 2025-03-29 |
| Document migration-first deploy process | @docs | 2025-03-22 |

Conduct postmortems for every SEV-1 and SEV-2 within 48 hours. Focus on system failures, not human errors. Ask "why did the system allow this?" Track action item completion. Review quarterly for recurring themes.

SLO Breach Handling

• Define SLOs (Service Level Objectives) for latency, availability, and error rate before incidents happen.
• Calculate error budgets: if SLO is 99.9% uptime, the error budget is 43.2 minutes per month.
• When the error budget is exhausted, freeze feature releases and focus on reliability work.
• Alert at 50% and 80% error budget consumption. Do not wait until the budget is gone.
• Tie SLO breaches to business impact. Leadership cares about customer trust, not percentiles.
• Review SLOs quarterly. Adjust targets based on customer expectations and operational capacity.
• Use SLO data to justify reliability investments. "We burned 90% of error budget last month" is a concrete argument.

Post-Incident Improvement

• Convert every postmortem action item into a tracked issue with a deadline and an owner.
• Prioritize automation that prevents recurrence over documentation that describes it.
• Add monitoring for the specific failure mode that caused the incident.
• Update runbooks with lessons learned from the incident.
• Share incident learnings in team retrospectives and engineering all-hands.
• Measure MTTR (Mean Time to Resolution) and MTTD (Mean Time to Detection) over time. Both should trend downward.

PagerDuty/OpsGenie Webhook Integration

// ✅ Production-Ready: PagerDuty Events API v2
import axios from "axios";

interface PagerDutyEvent {
  routing_key: string;
  event_action: "trigger" | "acknowledge" | "resolve";
  dedup_key?: string;
  payload: {
    summary: string;
    severity: "critical" | "error" | "warning" | "info";
    source: string;
    timestamp?: string;
    custom_details?: Record<string, unknown>;
  };
}

async function triggerPagerDutyAlert(
  routingKey: string,
  summary: string,
  severity: "critical" | "error" | "warning" | "info",
  details?: Record<string, unknown>
) {
  const event: PagerDutyEvent = {
    routing_key: routingKey,
    event_action: "trigger",
    payload: {
      summary,
      severity,
      source: "monitoring-system",
      timestamp: new Date().toISOString(),
      custom_details: details,
    },
  };

  const response = await axios.post(
    "https://events.pagerduty.com/v2/enqueue",
    event,
    { headers: { "Content-Type": "application/json" } }
  );

  return response.data.dedup_key; // Use this to acknowledge/resolve later
}

// Usage: trigger SEV-1 alert
const dedupKey = await triggerPagerDutyAlert(
  process.env.PAGERDUTY_ROUTING_KEY!,
  "Database connection pool exhausted",
  "critical",
  { activeConnections: 100, maxConnections: 100, queuedRequests: 250 }
);

# OpsGenie Alert API
import requests
import os

def create_opsgenie_alert(message: str, priority: str, details: dict):
    """Create OpsGenie alert via REST API."""
    url = "https://api.opsgenie.com/v2/alerts"
    headers = {
        "Authorization": f"GenieKey {os.getenv('OPSGENIE_API_KEY')}",
        "Content-Type": "application/json"
    }

    payload = {
        "message": message,
        "priority": priority,  # P1-P5
        "details": details,
        "tags": ["production", "automated"],
        "source": "monitoring"
    }

    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()
    return response.json()["requestId"]

# Usage
create_opsgenie_alert(
    message="API latency p99 > 2s for 5 minutes",
    priority="P1",
    details={"p99_latency": "2.3s", "endpoint": "/api/orders", "region": "us-east-1"}
)