bregman-arie When to use
Use for a confirmed high-severity production incident.
Preconditions
- You have an incident channel and a path to page additional responders.
Procedure
- Declare severity and open an incident channel.
- Assign roles: incident commander, comms lead, operations lead.
- State the customer impact in one sentence.
- Stabilize first: stop the bleeding (rollback, disable feature, shed load).
- Start a timeline and capture key actions/decisions.
Decision points
- If impact is unknown: prioritize measurement and narrowing scope.
- If rollback is low-risk and likely effective: rollback early.
- If data integrity is at risk: freeze writes and escalate.
Verification
- Leading indicators improve (error rate, latency, saturation).
- Customer reports decrease.
- Comms cadence is established.
Rollback / undo
- If mitigation worsens impact, revert and try the next lowest-risk action.
Escalation
- Escalate to security if incident involves auth, data exposure, or suspicious activity.
- Escalate to leadership if SLA/SLO breach is likely.
Examples
Initial update template:
Impact: <who/what is affected>
Start: <time>
Current status: <what we see>
Mitigation in progress: <what we are doing>
Next update: <time>