Containing an Incident
Stop the impact first and prove it stopped from the outside; leave the cause
to debugging.
When to use
- A failure is reaching users right now — requests erroring, a feature broken, data being corrupted or exposed, a queue backing up, a limit or budget being burned.
- Skip when nothing is reaching users: a red test, a bug found in review, a
defect nobody has hit. That is
debugging. - Skip once impact has stopped: an unknown technical cause goes to
debugging; a known cause and why safeguards missed it go topost-mortem. - If you are not sure whether users are affected, answering that is step 1, and it is fast. Do not resolve the doubt by starting to read code.
Step 1: Bound the impact
- Write four lines from the signal users are hitting, not from source: what fails, for whom, since when, how badly.
- Do not diagnose. Do not open product code.
Step 2: Pick the lever
List what could stop it in minutes, fastest and narrowest first:
- a feature flag, config value, or kill switch
- shedding or redirecting traffic, draining an instance, opening a breaker
- raising or lowering a limit, quota, or concurrency
- blocking the triggering input, tenant, or job
- rolling back to the last known-good release — wide, needs approval, unsafe once data was migrated
Read the runbook, deploy log, and config for them. Reading is not pulling.
A lever is yours only when the user, the incident owner, or a runbook the project designates gives you scoped authority for that lever on that target; holding credentials or access grants nothing. If a lever is not yours, escalate now with the Step 1 lines; that is the containment step. This includes every deploy, rollback, production-data change, and customer-facing block such as a tenant block, whatever access you hold.
If no safe authorized lever remains, or a lever risks evidence or other users, read
references/hard-containments.mdbefore anything else. Record the exhausted options, residual impact, and escalation as the record's hard-containment fallback; do not keep selecting the same lever.If capturing is cheap, and the user, the incident owner, or a designated runbook currently authorizes keeping that data, redact credentials and personal data, then save a failing example, the logs, or the current configuration to
.sdlc-skills/evidence/{{YYYY-MM-DD}}-{{topic}}/before a rollback or restart erases it. Seconds, not minutes. Never delay containment for evidence. If there is no such authority, keep nothing, and name what was lost in the record.Pull the narrowest lever that works. If its effect cannot be stated in one sentence, do not pull it. Open
assets/containment-record.mdwhen you pull it.Checkout fails for every user since a deploy that, the deploy log shows, put a new discount-code path behind a flag. Blocking the tenant that first reported it is the narrowest scope but does not work, since every user is failing; a rollback works but is wide. Turning the flag off is the narrowest lever that works, and its effect fits one sentence: checkout runs without discount codes. Pull it if Step 2.3 makes it yours.
Step 3: Prove it stopped
- Read the signal from Step 1 again. If it has returned to normal, the impact is contained.
- If the signal is unchanged, the impact is not contained. Record the attempted lever, its result, and whether it still holds; return to Step 2 only for a different justified intervention. If the options are exhausted, take Step 2.4's hard-containment fallback. The action taken proves nothing.
- Any other signal (partial, delayed, unreadable, returning): read
references/hard-containments.mdbefore deciding.
Step 4: Record and hand off
Complete the containment record from Step 2.6. Open
assets/containment-record.mdhere when no lever was pulled, because Step 2.3 escalated or Step 2.4 exhausted the options, and put the escalation in its lever table.Report three things and call nothing resolved:
Stopped: {{what the signal shows}} Still true: {{the feature is still broken}} Cost while it holds: {{what the mitigation costs}}REQUIRED SUB-SKILL: invoke
debuggingfor the cause. Once cause and containment are known, invokepost-mortem. Write both into the record'sStill openlines. Diagnose nothing from this skill.Pass any hard-containment fallback to
debugging. New or worsened impact or a newly available lever reopens containment; unchanged residual impact does not.
Gotchas
- Access that can pull a lever is not authority to pull it: an unauthorized rollback or tenant block becomes a second incident.
When you are tempted to skip it
| The thought | The reality |
|---|---|
| "Five more minutes and I'll have the real fix" | Five minutes of certainty is five minutes of users failing. The lever costs seconds and reverses. |
| "I found the cause — a proper fix beats a flag" | A correct fix still has to be reviewed, tested, and shipped. The flag is live now. |
| "I read the runbook, so I know the lever is there" | Reading a lever is not pulling it. |
| "Rolling back drops the other changes in that release" | Then use a narrower lever. Blast radius is a reason to choose better, not a reason to do nothing. |
| "The tests are green, so it can't be that bad" | The suite covers what someone thought of. The users are the evidence. |
| "I don't have deploy access" | Then escalating is the containment step, and it starts now. |
| "Flipping the flag just hides the bug" | It hides it from users. It stays visible in the branch, the tests, and the record. |
| "It's been broken for an hour already" | Every minute from now is one you chose. |
| "It only affects some requests" | Then contain for those. Partial impact is impact. |
| "Containment worked, so we're done" | The mitigation is load-bearing and the feature is still broken. |
Red flags
Without a recorded hard-containment fallback, stop and go to step 2 for these:
- You are reading source code and users are still failing.
- You know which flag or release would stop it, and it is still not pulled.
- Your first edit is in the product code.
- You are writing a reproduction for a failure that is reproducing itself in production.
- You are explaining the cause in your reply while the impact is ongoing.
- The mitigation is in place and nothing records that it is.