Raz-Gits blast radius
Most engineering advice is about correctness. This is about the other thing: what happens when the code is wrong, which it will be, and how much of the world it touches before anybody finds out.
The question is not "how much can this touch". It is:
With the worst plausible input, how much does this touch before someone notices?
Those are different numbers, and the gap between them is the detection window. Most incidents are not caused by especially bad code. They are caused by ordinary code running unattended for longer than anyone expected.
When this applies
Anything with a side effect outside the process: sending email or SMS, writing to a CRM or a production database, charging or refunding, posting publicly, deleting, deploying, or any scheduled job that calls a third party.
It does not apply to pure computation. Do not slow down a report script with this.
The six questions
Answer these before it runs the first time. If any answer is "I don't know", that is the finding.
1. Can it run without doing anything?
If you cannot run it in report-only mode, you do not yet understand what it does. That is the real value of the mode, more than the safety: building it forces you to separate deciding from acting, and the code is better afterwards regardless.
Report-only should be the default, with acting behind an explicit flag. The opposite arrangement means that every future mistake, every bad config, every half-finished branch, sends for real.
Print exactly what it would have done, in enough detail to check. "Would send 412 emails" is not enough. "Would send to these 412 addresses, here are the first twenty" is.
2. What happens if it runs twice?
Retries happen. Crons overlap. Someone reruns a failed job. A webhook provider redelivers.
Either the operation is idempotent, or something durable prevents the second run from repeating the first. A key, a ledger, a lock. "It won't run twice" is not an answer, it is an assumption, and it is the assumption that produces the story about the customer who got the same email nine times.
Overlap on schedules deserves its own thought. Two copies of the same job racing each other is the common version of this.
3. Where does the human go, and is it worth it?
A gate on everything is paralysis and people route around it, which is worse than no gate because now you have a gate you believe in and nobody uses.
The gate earns its place where the cost is asymmetric: a wrong action costs more than reviewing every action. Sending to the wrong list, charging the wrong card, posting under someone's name, deleting. Those are worth a human. Fetching, classifying, drafting, scoring are not.
When it is worth it, put the gate at the last reversible moment, not the first. Gating the fetch teaches nothing. Gating the send is the point.
And review has to be possible. A person approving 400 items will approve 400 items. If the queue is that long, the gate has become theatre and the honest fix is to make the automation good enough not to need one, or to sample.
4. Is it reversible, and what is the smallest irreversible unit?
Some things undo. A CRM field reverts. A deploy rolls back.
Some do not. A sent email is sent. A charge is a refund plus a support ticket, not an undo. A public post was seen.
For the irreversible ones, the question becomes: what is the smallest batch you can act in and still check between batches? Ten sends, then look, then the rest. The whole list at once is a choice, and it should be a deliberate one.
5. Who must never be touched, and is that list authoritative?
Every system that acts on people needs an exclusion list: opted out, deleted, already contacted, do not contact, test accounts, internal staff.
Two properties matter more than the list itself. It must be checked at the last moment before acting, not at the start of the run, because runs are long and people opt out during them. And it must be append-only, because an exclusion that can be edited or rebuilt can quietly resurrect someone who asked to be left alone. That is the one mistake in this whole document with legal consequences.
6. How will you know it broke?
The worst failure is not the crash. A crash is loud and someone fixes it. The worst failure is the job that exits zero and does nothing, because everything looks healthy for as long as nobody checks.
A non-zero exit catches crashes. Nothing catches silence except something that expects to hear from the job:
- a heartbeat written on success, and an alert when the heartbeat goes stale
- a floor on the expected result, so "zero rows today" is a failure rather than a quiet Tuesday
- comparison against yesterday, because a collapse from 4,000 to 6 is not a crash
A system that is right ninety-five percent of the time and screams when it is wrong beats one that is right ninety-nine percent and silent. The second one teaches you to trust it, and then spends the trust all at once.
The write-up
Short. Six answers and a number.
Blast radius: <what this touches>
Worst plausible input: <what it does then>
Detection window: <how long before someone notices>
Report-only: yes / no, and why
Idempotent: yes / by what mechanism / no
Human gate: where, or why none
Reversible: yes / no, smallest batch is <n>
Exclusions: <list>, checked <when>
Alarm: <what fires, and what makes it fire>
If the detection window is measured in days and the blast radius is measured in customers, that pairing is the finding. Say it plainly rather than burying it.
Rules
- Report-only is the default, not the option. Every argument for the reverse is an argument about convenience, made before the incident.
- Never grant an exception under time pressure. "Just this once, skip the dry run" is the sentence that precedes most of these stories.
- Do not confuse tested with safe. Tests prove the code does what you meant. Blast radius is about the case where what you meant was wrong.
- Name the unknown rather than assuming the good case. "I don't know whether the provider deduplicates" is a useful sentence. Assuming it does is not.
- Do not apply this to things that cannot hurt anyone. Ceremony on a read-only script trains people to ignore the ceremony where it counts.