Incident Response
Manage active production incidents from detection to resolution. Stack-agnostic. Tool-agnostic.
This skill is for active incidents and incident process. For after-the-fact analysis, use after-action-report. For planned launches, use launch-runbook.
When to use
- An active incident is happening
- Building incident response procedures
- Defining severity levels
- Setting up on-call rotations
- Training a team on incident response
When NOT to use
- Post-incident retrospective (use
after-action-report)
- Planned launches (use
launch-runbook)
- Pre-launch issue triage (use
qa-testing)
Required inputs
- Awareness of the incident (alert, customer report, internal observation)
- Access to production systems and monitoring
- Roles and authorities clearly defined
- Communication channels operational
The framework: 5 phases
1. Detection
How the incident becomes known.
Detection sources:
- Automated alerts (monitoring, SLO violations, error rate spikes)
- Customer reports (support tickets, social media, status page subscribers)
- Internal observation (engineer notices something off)
- Third-party (security researchers, partners)
On detection:
- Acknowledge within target time (typically 5 to 15 minutes for critical)
- Assess severity (see severity rubric below)
- Page the on-call if not already paged
- Open the incident channel
2. Triage
Establish severity and impact.
Severity rubric:
| Severity |
Definition |
Response |
| SEV-1 (Critical) |
Major customer-facing functionality broken. Data integrity at risk. Security breach. |
All-hands. Incident commander. Active war room. Public communication required. |
| SEV-2 (Major) |
Significant degradation. Some customers affected. Revenue impact. |
Incident commander assigned. Active response. Internal communication. May or may not need public communication. |
| SEV-3 (Minor) |
Limited impact. Workaround available. Affecting a small group of users. |
Standard on-call response. Single owner. |
| SEV-4 (Low) |
Cosmetic, edge-case, or low-frequency. No urgent action needed. |
Tracked as bug. Addressed in normal queue. |
Severity can change. Re-evaluate as more info emerges.
3. Mitigation
Stop the bleeding before fixing the cause.
Mitigation patterns (faster than full fix):
- Rollback (revert recent deploy)
- Feature flag off (disable the broken feature without deploy)
- Failover (route to healthy replica or region)
- Scale up (more capacity to absorb the load)
- Throttle (reject some traffic to protect the rest)
- Graceful degradation (turn off non-essential features to keep core functional)
- Maintenance mode (last resort, blocks all users)
Mitigation principle: Stop user impact first. Cause analysis second.
4. Communication
Three audiences during an incident:
Internal team:
- Real-time updates in incident channel
- Cadence: every 15 minutes minimum during active incident
- Format: timestamped status updates with what we know, what we're doing, ETA
Internal stakeholders:
- Higher-level updates to broader org
- Cadence: every 30 to 60 minutes
- Format: business-impact framing, not technical detail
External / customers:
- Status page updates
- Cadence: every 30 minutes minimum during active incident
- Format: plain language, no blame, what users are experiencing, what to expect
Communication principles:
- Acknowledge before you have answers ("We're aware and investigating")
- Update on schedule even if no progress ("Still investigating, no new information")
- Never speculate publicly about cause
- Confirm resolution explicitly when restored
5. Resolution
Verified fix, customers restored, incident closed.
Resolution criteria:
- Mitigation in place and verified
- Root cause identified (or explicitly deferred to AAR)
- All affected systems back to normal
- Customers can resume normal use
- Final status update posted (internal and external)
- Incident channel can be closed (or archived for AAR)
After closure:
- Schedule AAR within 1 to 2 weeks
- Capture initial timeline while memories are fresh
- Track follow-up action items
Roles during an incident
| Role |
Responsibility |
| Incident commander (IC) |
Owns the response. Calls decisions. Assigns work. Not necessarily the most technical person; needs to coordinate. |
| Communications lead |
Owns internal and external messaging. Reduces IC's communication burden. |
| Operations lead |
Drives the technical investigation and mitigation. Often the most senior on-call engineer. |
| Scribe |
Captures the timeline as the incident unfolds. Critical for AAR. |
| Subject matter experts |
Pulled in as needed. Service owners, database experts, security experts. |
For small teams or low-severity incidents, one person can hold multiple roles. Each role's responsibilities should still be explicit.
Decision-making during an incident
The IC's authority:
- Call rollback or other mitigations
- Pull additional people in
- Escalate severity
- Make the call when unclear options exist
Non-decisions to avoid:
- "Let's wait and see" when mitigations are available and impact is occurring
- Discussing root cause while users are actively impacted (mitigate first)
- Premature resolution announcements before verification
- Death-by-committee (pull in lots of people, no one decides)
When in doubt: act. A wrong action that can be rolled back beats inaction while users suffer.
Status page communication patterns
Initial:
"We are investigating reports of [issue]. Updates to follow."
Identified:
"We have identified the issue affecting [scope]. Engineers are working on a fix. Next update by [time]."
Monitoring:
"A fix has been applied. We are monitoring to confirm resolution. Next update by [time]."
Resolved:
"This incident has been resolved. Service has been restored. A full incident report will be posted within [timeframe]."
Patterns to avoid:
- Vague language ("experiencing some issues" - what kind?)
- Missing affected scope ("login is down" - everywhere or just one region?)
- Missing time commitments
- "Should be resolved soon" without verification
- Using "back up" before verification
Workflow
- Acknowledge. First responder acknowledges within target time.
- Assess severity. Use the rubric. Open the appropriate response channel.
- Assign roles. IC, comms, ops at minimum.
- Communicate. Initial status update. Internal channel active.
- Investigate. Logs, metrics, recent changes. The four most common causes: a recent deploy, a configuration change, a third-party dependency change, a load spike.
- Mitigate. Stop the bleeding. Don't wait for full root cause.
- Verify mitigation. Don't trust dashboards alone; test the user flow.
- Communicate resolution. Internal and external.
- Close incident. Final timeline noted. Action items tracked.
- Schedule AAR. Within 1 to 2 weeks.
Failure patterns
- No clear IC. Multiple people debugging in parallel, no coordination. Slower to mitigate, easier to make conflicting changes.
- Skipping mitigation, going straight to root cause. Users keep suffering while engineers debug.
- Premature "all clear." Announcing resolution before verification.
- Communication silence. Users don't know if anyone is working on it.
- Status updates too vague. "We're working on it" with no detail.
- Speculating publicly about cause. Often wrong, always damaging trust.
- Pulling in too many people. Coordination overhead exceeds value.
- No scribe. The timeline gets lost. AAR has to reconstruct from chat logs.
- Skipping AAR for "minor" incidents. Patterns get missed. Lessons get re-learned.
- Blame culture. People hide mistakes, incidents take longer.
Output format
During an active incident: incident channel updates and status page updates as per the framework above.
After incident close: a brief incident summary feeding into the AAR.
# Incident: [Brief title]
**Date:** [YYYY-MM-DD]
**Severity:** [SEV-1 / 2 / 3 / 4]
**Duration:** [Detection to resolution]
**Customer impact:** [Who, how many, how, or state the gap per the data-availability rule]
## Summary
[1 to 2 paragraphs]
## Timeline
[Timestamped events]
## Mitigation
[What was done]
## Action items
[Follow-ups, with owners]
## AAR scheduled for
[Date]
If required data is unavailable
This skill's output depends on data, measurements, or tool results it cannot generate on its own. When a required input, tool, or data source is unavailable or unverifiable, the sanctioned output is the deliverable with the gap stated: what was needed, what was actually obtained or verified, and which parts of the output are affected. Fabricating, estimating, or interpolating a required number to complete the deliverable is never sanctioned. A stated gap is a complete answer.
Reference files
references/incident-playbook.md - Severity definitions, roles, status page templates, decision rubrics.
1---2name: "incident-response"3description: "Manage active production incidents through detection, triage, mitigation, communication, and resolution with structured roles and decision-making. Use this skill whenever the user has an active incident, a production issue, a service outage, a security incident, or needs to plan incident response procedures. Triggers on incident response, production incident, outage, service down, site down, P0, P1, severity, downtime, on-call, incident commander, status page, postmortem prep. Also triggers when something is actively broken in production and the user is figuring out what to do."4metadata:5 source: https://skillmd.com/skills/slipcast-dev/incident-response6---78# Incident Response910Manage active production incidents from detection to resolution. Stack-agnostic. Tool-agnostic.1112This skill is for active incidents and incident process. For after-the-fact analysis, use `after-action-report`. For planned launches, use `launch-runbook`.1314---1516## When to use1718- An active incident is happening19- Building incident response procedures20- Defining severity levels21- Setting up on-call rotations22- Training a team on incident response2324## When NOT to use2526- Post-incident retrospective (use `after-action-report`)27- Planned launches (use `launch-runbook`)28- Pre-launch issue triage (use `qa-testing`)2930---3132## Required inputs3334- Awareness of the incident (alert, customer report, internal observation)35- Access to production systems and monitoring36- Roles and authorities clearly defined37- Communication channels operational3839---4041## The framework: 5 phases4243### 1. Detection4445How the incident becomes known.4647**Detection sources:**4849- Automated alerts (monitoring, SLO violations, error rate spikes)50- Customer reports (support tickets, social media, status page subscribers)51- Internal observation (engineer notices something off)52- Third-party (security researchers, partners)5354**On detection:**5556- Acknowledge within target time (typically 5 to 15 minutes for critical)57- Assess severity (see severity rubric below)58- Page the on-call if not already paged59- Open the incident channel6061### 2. Triage6263Establish severity and impact.6465**Severity rubric:**6667| Severity | Definition | Response |68|---|---|---|69| SEV-1 (Critical) | Major customer-facing functionality broken. Data integrity at risk. Security breach. | All-hands. Incident commander. Active war room. Public communication required. |70| SEV-2 (Major) | Significant degradation. Some customers affected. Revenue impact. | Incident commander assigned. Active response. Internal communication. May or may not need public communication. |71| SEV-3 (Minor) | Limited impact. Workaround available. Affecting a small group of users. | Standard on-call response. Single owner. |72| SEV-4 (Low) | Cosmetic, edge-case, or low-frequency. No urgent action needed. | Tracked as bug. Addressed in normal queue. |7374Severity can change. Re-evaluate as more info emerges.7576### 3. Mitigation7778Stop the bleeding before fixing the cause.7980**Mitigation patterns (faster than full fix):**8182- **Rollback** (revert recent deploy)83- **Feature flag off** (disable the broken feature without deploy)84- **Failover** (route to healthy replica or region)85- **Scale up** (more capacity to absorb the load)86- **Throttle** (reject some traffic to protect the rest)87- **Graceful degradation** (turn off non-essential features to keep core functional)88- **Maintenance mode** (last resort, blocks all users)8990**Mitigation principle:** Stop user impact first. Cause analysis second.9192### 4. Communication9394Three audiences during an incident:9596**Internal team:**97- Real-time updates in incident channel98- Cadence: every 15 minutes minimum during active incident99- Format: timestamped status updates with what we know, what we're doing, ETA100101**Internal stakeholders:**102- Higher-level updates to broader org103- Cadence: every 30 to 60 minutes104- Format: business-impact framing, not technical detail105106**External / customers:**107- Status page updates108- Cadence: every 30 minutes minimum during active incident109- Format: plain language, no blame, what users are experiencing, what to expect110111**Communication principles:**112- Acknowledge before you have answers ("We're aware and investigating")113- Update on schedule even if no progress ("Still investigating, no new information")114- Never speculate publicly about cause115- Confirm resolution explicitly when restored116117### 5. Resolution118119Verified fix, customers restored, incident closed.120121**Resolution criteria:**122123- Mitigation in place and verified124- Root cause identified (or explicitly deferred to AAR)125- All affected systems back to normal126- Customers can resume normal use127- Final status update posted (internal and external)128- Incident channel can be closed (or archived for AAR)129130After closure:131- Schedule AAR within 1 to 2 weeks132- Capture initial timeline while memories are fresh133- Track follow-up action items134135---136137## Roles during an incident138139| Role | Responsibility |140|---|---|141| Incident commander (IC) | Owns the response. Calls decisions. Assigns work. Not necessarily the most technical person; needs to coordinate. |142| Communications lead | Owns internal and external messaging. Reduces IC's communication burden. |143| Operations lead | Drives the technical investigation and mitigation. Often the most senior on-call engineer. |144| Scribe | Captures the timeline as the incident unfolds. Critical for AAR. |145| Subject matter experts | Pulled in as needed. Service owners, database experts, security experts. |146147For small teams or low-severity incidents, one person can hold multiple roles. Each role's responsibilities should still be explicit.148149---150151## Decision-making during an incident152153**The IC's authority:**154155- Call rollback or other mitigations156- Pull additional people in157- Escalate severity158- Make the call when unclear options exist159160**Non-decisions to avoid:**161162- "Let's wait and see" when mitigations are available and impact is occurring163- Discussing root cause while users are actively impacted (mitigate first)164- Premature resolution announcements before verification165- Death-by-committee (pull in lots of people, no one decides)166167When in doubt: act. A wrong action that can be rolled back beats inaction while users suffer.168169---170171## Status page communication patterns172173**Initial:**174> "We are investigating reports of [issue]. Updates to follow."175176**Identified:**177> "We have identified the issue affecting [scope]. Engineers are working on a fix. Next update by [time]."178179**Monitoring:**180> "A fix has been applied. We are monitoring to confirm resolution. Next update by [time]."181182**Resolved:**183> "This incident has been resolved. Service has been restored. A full incident report will be posted within [timeframe]."184185Patterns to avoid:186187- Vague language ("experiencing some issues" - what kind?)188- Missing affected scope ("login is down" - everywhere or just one region?)189- Missing time commitments190- "Should be resolved soon" without verification191- Using "back up" before verification192193---194195## Workflow1961971. **Acknowledge.** First responder acknowledges within target time.1982. **Assess severity.** Use the rubric. Open the appropriate response channel.1993. **Assign roles.** IC, comms, ops at minimum.2004. **Communicate.** Initial status update. Internal channel active.2015. **Investigate.** Logs, metrics, recent changes. The four most common causes: a recent deploy, a configuration change, a third-party dependency change, a load spike.2026. **Mitigate.** Stop the bleeding. Don't wait for full root cause.2037. **Verify mitigation.** Don't trust dashboards alone; test the user flow.2048. **Communicate resolution.** Internal and external.2059. **Close incident.** Final timeline noted. Action items tracked.20610. **Schedule AAR.** Within 1 to 2 weeks.207208---209210## Failure patterns211212- **No clear IC.** Multiple people debugging in parallel, no coordination. Slower to mitigate, easier to make conflicting changes.213- **Skipping mitigation, going straight to root cause.** Users keep suffering while engineers debug.214- **Premature "all clear."** Announcing resolution before verification.215- **Communication silence.** Users don't know if anyone is working on it.216- **Status updates too vague.** "We're working on it" with no detail.217- **Speculating publicly about cause.** Often wrong, always damaging trust.218- **Pulling in too many people.** Coordination overhead exceeds value.219- **No scribe.** The timeline gets lost. AAR has to reconstruct from chat logs.220- **Skipping AAR for "minor" incidents.** Patterns get missed. Lessons get re-learned.221- **Blame culture.** People hide mistakes, incidents take longer.222223---224225## Output format226227During an active incident: incident channel updates and status page updates as per the framework above.228229After incident close: a brief incident summary feeding into the AAR.230231```markdown232# Incident: [Brief title]233234**Date:** [YYYY-MM-DD]235**Severity:** [SEV-1 / 2 / 3 / 4]236**Duration:** [Detection to resolution]237**Customer impact:** [Who, how many, how, or state the gap per the data-availability rule]238239## Summary240[1 to 2 paragraphs]241242## Timeline243[Timestamped events]244245## Mitigation246[What was done]247248## Action items249[Follow-ups, with owners]250251## AAR scheduled for252[Date]253```254255---256257## If required data is unavailable258259This skill's output depends on data, measurements, or tool results it cannot generate on its own. When a required input, tool, or data source is unavailable or unverifiable, the sanctioned output is the deliverable with the gap stated: what was needed, what was actually obtained or verified, and which parts of the output are affected. Fabricating, estimating, or interpolating a required number to complete the deliverable is never sanctioned. A stated gap is a complete answer.260261---262263## Reference files264265- [`references/incident-playbook.md`](references/incident-playbook.md) - Severity definitions, roles, status page templates, decision rubrics.266