Incident Response
An incident is a coordination problem before it is a debugging problem. Several people (and agents)
act at once, facts arrive out of order, and users want to know what is happening. This skill gives
the response a shape: one severity, named roles, one timeline, mitigation first, and a review that
turns the outage into tracked work.
When to Use
- An alert, support report, or monitor shows users are failing, data is wrong, or a security
boundary may be breached
- A deploy, migration, or config change is suspected of causing live harm
- More than one person or agent is already working the problem and nobody owns coordination
- After recovery, to run the post-incident review and record follow-up actions
When NOT to Use
| Instead of incident-response |
Use |
| Watching a release that is still healthy behind a canary |
deployment-canary |
| Finding the root cause once users are no longer affected |
systematic-debugging |
| A failing test or CI job with no user impact |
actions-debugging or fix-build-errors |
| A suspected vulnerability with no evidence of exploitation |
security-scan or threat-model-analyst |
Prerequisites
- A place everyone can read: an incident issue, a chat channel, or a shared document
- Access to the signals that show user impact (dashboards, logs, error tracker, status checks)
- Knowledge of the rollback or failover paths for the affected system
Workflow
1. Declare and Assign Roles
Declare the incident as soon as impact is likely; downgrading later is cheap, declaring late is not.
Record a severity using the team's scale, or this default:
| Severity |
Meaning |
| SEV1 |
Core workflow down or data loss/exposure for many users |
| SEV2 |
Major feature degraded, workaround exists, or a subset of users affected |
| SEV3 |
Minor impact, no urgency beyond business hours |
Name three roles, even if one person holds two:
- Incident lead — owns decisions and priorities; does not debug
- Operations lead — runs the technical investigation and changes
- Communications — writes status updates and keeps the timeline
Open the record with the GitHub CLI when the team tracks incidents as issues (the incident label
must already exist — create it once with gh label create incident):
gh issue create --title "SEV2: checkout errors after 14:05 deploy" --label incident `
--body "Lead: @a Ops: @b Comms: @c Status: investigating"
2. Keep One Timeline
Append every observation and action with a timestamp in one place. Write facts, not conclusions:
"14:12 error rate 8% on /checkout (dashboard X)" rather than "14:12 DB is overloaded". When an agent
runs commands, record the command, the time, and the output that drove a decision. Redact
credentials, personal data, and customer data; keep large raw output in an access-controlled place
and link to it from the timeline.
3. Mitigate Before You Diagnose
Stop the user impact first, with the least risky reversible action:
- Roll back the suspected change, or turn off its feature flag
- Fail over, shed load, or scale the saturated component
- Block the abusive or malformed traffic
Each mitigation is a change: state it in the timeline before running it, and say how you will know
whether it worked. Root-cause work starts once impact is contained — hand it to
systematic-debugging with the timeline attached.
4. Communicate on a Fixed Cadence
Post status at a fixed interval (for example every 30 minutes for SEV1, 60 for SEV2), even when
nothing changed. Each update states: current impact, what is being done, and the time of the next
update. Do not speculate about cause in external updates.
5. Resolve and Review
Close the incident only after the recovery checks below pass. Within a few days, run a blameless
review:
- Timeline of detection, mitigation, and recovery, with times
- What made detection or mitigation slow
- Contributing causes (usually several), not a single culprit
- Actions, each with an owner and a due date, filed as issues
Common Rationalizations
| Rationalization |
Reality |
| "It's probably nothing, let's look first" |
Late declaration is the most common reason incidents run long |
| "I'll fix the root cause, that's faster than rolling back" |
A fix under pressure is an untested change; roll back first |
| "No new information, so no update" |
Silence reads as abandonment; "no change, next update 15:30" is an update |
| "We know whose change caused it" |
Blame stops people reporting; contributing causes stop recurrence |
Red Flags
- Nobody can say who the incident lead is
- Several people are making production changes without writing them down
- The status update promises a cause or an ETA the team does not have
- The review produces "be more careful" instead of owned, dated actions
Verification
If recovery checks fail after a mitigation, reopen the response at step 3 with the next mitigation
rather than declaring partial success.
Tips
- Agents are useful as scribes and investigators: give one the timeline role and others bounded
read-only investigations, and keep production changes with the operations lead
- Use the review actions as input to
deployment-canary rollback triggers for the next release
1---2name: incident-response3license: MIT4description: Use when production users are affected or data integrity is at risk and someone must run the response — declare severity and roles, keep one timeline, separate mitigation from root cause, post status on a fixed cadence, and close with a blameless review whose actions have owners.5metadata:6 source: https://skillmd.com/skills/drvoss/incident-response7 category: workflow8 agent_type: general-purpose9---1011# Incident Response1213An incident is a coordination problem before it is a debugging problem. Several people (and agents)14act at once, facts arrive out of order, and users want to know what is happening. This skill gives15the response a shape: one severity, named roles, one timeline, mitigation first, and a review that16turns the outage into tracked work.1718## When to Use1920- An alert, support report, or monitor shows users are failing, data is wrong, or a security21 boundary may be breached22- A deploy, migration, or config change is suspected of causing live harm23- More than one person or agent is already working the problem and nobody owns coordination24- After recovery, to run the post-incident review and record follow-up actions2526## When NOT to Use2728| Instead of incident-response | Use |29|------------------------------|-----|30| Watching a release that is still healthy behind a canary | `deployment-canary` |31| Finding the root cause once users are no longer affected | `systematic-debugging` |32| A failing test or CI job with no user impact | `actions-debugging` or `fix-build-errors` |33| A suspected vulnerability with no evidence of exploitation | `security-scan` or `threat-model-analyst` |3435## Prerequisites3637- A place everyone can read: an incident issue, a chat channel, or a shared document38- Access to the signals that show user impact (dashboards, logs, error tracker, status checks)39- Knowledge of the rollback or failover paths for the affected system4041## Workflow4243### 1. Declare and Assign Roles4445Declare the incident as soon as impact is likely; downgrading later is cheap, declaring late is not.46Record a severity using the team's scale, or this default:4748| Severity | Meaning |49|----------|---------|50| SEV1 | Core workflow down or data loss/exposure for many users |51| SEV2 | Major feature degraded, workaround exists, or a subset of users affected |52| SEV3 | Minor impact, no urgency beyond business hours |5354Name three roles, even if one person holds two:5556- **Incident lead** — owns decisions and priorities; does not debug57- **Operations lead** — runs the technical investigation and changes58- **Communications** — writes status updates and keeps the timeline5960Open the record with the GitHub CLI when the team tracks incidents as issues (the `incident` label61must already exist — create it once with `gh label create incident`):6263```powershell64gh issue create --title "SEV2: checkout errors after 14:05 deploy" --label incident `65 --body "Lead: @a Ops: @b Comms: @c Status: investigating"66```6768### 2. Keep One Timeline6970Append every observation and action with a timestamp in one place. Write facts, not conclusions:71"14:12 error rate 8% on /checkout (dashboard X)" rather than "14:12 DB is overloaded". When an agent72runs commands, record the command, the time, and the output that drove a decision. Redact73credentials, personal data, and customer data; keep large raw output in an access-controlled place74and link to it from the timeline.7576### 3. Mitigate Before You Diagnose7778Stop the user impact first, with the least risky reversible action:79801. Roll back the suspected change, or turn off its feature flag812. Fail over, shed load, or scale the saturated component823. Block the abusive or malformed traffic8384Each mitigation is a change: state it in the timeline before running it, and say how you will know85whether it worked. Root-cause work starts once impact is contained — hand it to86`systematic-debugging` with the timeline attached.8788### 4. Communicate on a Fixed Cadence8990Post status at a fixed interval (for example every 30 minutes for SEV1, 60 for SEV2), even when91nothing changed. Each update states: current impact, what is being done, and the time of the next92update. Do not speculate about cause in external updates.9394### 5. Resolve and Review9596Close the incident only after the recovery checks below pass. Within a few days, run a blameless97review:9899- Timeline of detection, mitigation, and recovery, with times100- What made detection or mitigation slow101- Contributing causes (usually several), not a single culprit102- Actions, each with an owner and a due date, filed as issues103104## Common Rationalizations105106| Rationalization | Reality |107|-----------------|---------|108| "It's probably nothing, let's look first" | Late declaration is the most common reason incidents run long |109| "I'll fix the root cause, that's faster than rolling back" | A fix under pressure is an untested change; roll back first |110| "No new information, so no update" | Silence reads as abandonment; "no change, next update 15:30" is an update |111| "We know whose change caused it" | Blame stops people reporting; contributing causes stop recurrence |112113## Red Flags114115- Nobody can say who the incident lead is116- Several people are making production changes without writing them down117- The status update promises a cause or an ETA the team does not have118- The review produces "be more careful" instead of owned, dated actions119120## Verification121122- [ ] The user-facing signal that showed the impact is back to its baseline, not just a component123 health check124- [ ] The mitigation's side effects are checked (rolled-back data, disabled features, queued work)125- [ ] Any data written during the incident is confirmed correct, or a repair action is filed126- [ ] The timeline is complete enough for someone absent to follow the response127- [ ] Every review action has an owner, a due date, and an issue128129If recovery checks fail after a mitigation, reopen the response at step 3 with the next mitigation130rather than declaring partial success.131132## Tips133134- Agents are useful as scribes and investigators: give one the timeline role and others bounded135 read-only investigations, and keep production changes with the operations lead136- Use the review actions as input to `deployment-canary` rollback triggers for the next release