Incident Response

Incident Response

When production is failing, the first job is containment, not root cause. This skill structures the live-incident workflow: classify severity, stop the bleeding, communicate, then diagnose — the inverse of debugging, which builds a red loop first. A blameless postmortem follows every incident so the system improves, not just the symptom.

When to use

• A production incident is live (page fired, users affected, error spike)
• An incident just resolved and needs a postmortem
• Reviewing or rehearsing incident response readiness

Not for: diagnosing a dev-time bug (use debugging); adding telemetry after the fact (use observability); launching a new release (use shipping). This skill is for when something already running has broken.

Steps

1. Declare and classify severity

Assign a severity within the first 5 minutes — severity drives everything downstream (who wakes up, how fast you respond, what comms are required). Use a fixed scale, not ad-hoc:

• SEV1 — user-facing outage or data loss. Page everyone. Incident commander assigned.
• SEV2 — significant degradation, partial outage. Page on-call. IC assigned.
• SEV3 — minor degradation, workaround exists. Ticket, address in business hours.

Record the declaration: start time, severity, initial symptom, who is IC. Verify: severity is written down with a timestamp, not just in someone's head.

2. Contain (stop the bleeding)

The first decision is containment vs. fix. Containment is faster and safer — rollback, disable the offending feature flag, route around the failing dependency, shed load. A fix under pressure is a second incident waiting to happen.

• Rollback to the last known-good deploy (fastest). Platform commands: kubectl rollout undo deployment/<name> (Kubernetes), aws deploy create-deployment --revision REVISION<prev> (AWS CodeDeploy), gcloud run services update-traffic --to-revisions (Cloud Run), vercel rollback <url> (Vercel). The shipping skill documents when to roll back (trigger conditions); these are the how commands for a live SEV1.
• Disable the feature flag that triggered the issue
• Fail over to a healthy replica or region
• Rate-limit or circuit-break the failing path

Only attempt a forward fix if rollback is impossible (irreversible migration, no prior deploy) and the fix is small and obvious. Verify: the bleeding has stopped — error rate dropping or users recovering — before moving to diagnosis.

3. Communicate

The IC owns comms. Establish a single incident channel and a cadence:

• Internal: status update every 15–30 min (SEV1) or hourly (SEV2): current state, what's been tried, next step, owner, ETA. Stale silence breeds panic and duplicate work.
• External: status page update for user-facing incidents — users tolerate outage, they don't tolerate silence. Update at the same cadence as internal.
• Stakeholders: notify leadership for SEV1; keep them informed, not involved in the fix.

Verify: the last update is less than 30 min old and states the next action with an owner.

4. Diagnose (after containment)

With the bleeding stopped, find root cause. Apply debugging discipline: build a hypothesis, instrument, verify — but under the incident's time pressure, favor the fastest path to a confirmed cause over exhaustive analysis. Use the telemetry observability built: traces, metrics, logs, correlation IDs. If the diagnosis stalls, contain harder and hand off to a fresh responder.

Verify: root cause is stated as a confirmed hypothesis with supporting evidence (log line, metric spike, deploy diff), not a guess.

5. Fix and verify

Apply the smallest fix that addresses the root cause. Verify at runtime — error rate returned to baseline, affected users recovered, no new regression. Re-enable any containment measures disabled (feature flags, failover) only after the fix is confirmed.

Verify: production metrics are at baseline for a sustained window (not a single point) before declaring resolved.

6. Postmortem (blameless)

Within 48 hours, write a blameless postmortem. Blameless means: focus on the system and process, not individuals — "the deploy bypassed the canary" not "X deployed without checking." Every contributing factor is a system gap, not a personal failure.

• Timeline: minute-by-minute from detection to resolution
• Impact: users affected, duration, data loss, revenue
• Root cause: the confirmed cause from step 4
• Contributing factors: what made it possible (missing test, no alert, manual step)
• What went well: detection time, rollback worked, comms cadence held
• Action items: specific, owned, dated — each addresses a contributing factor

Output: docs/postmortem/YYYY-MM-DD-<slug>.md — one file per incident, archived for organizational learning.

Verify

• Severity declared within 5 min, written down with timestamp
• Containment chosen over forward-fix unless rollback was impossible
• Comms cadence established; last update <30 min old during the incident
• Root cause stated with evidence (not a guess)
• Production metrics at baseline for a sustained window before "resolved"
• Postmortem written within 48 hours, blameless, with owned action items

Red flags: diagnosing before containing; no written severity declaration; silent incident channel; forward-fixing under pressure when rollback was available; postmortem that names individuals instead of system gaps; action items with no owner or date.

References

• ${CLAUDE_PLUGIN_ROOT}/references/engineering-principles.md — shared discipline (verify don't assume, surface assumptions)
• references/runbook-template.md — runbook format for alert-linked playbooks
• references/postmortem-template.md — blameless postmortem template + action-item tracking
• references/dr-planning.md — load during postmortem action-item planning for proactive DR; backup strategy, restore testing, DR drills, failover orchestration