Accuracy is not savings
The model is 92% accurate, so it removes 92% of the work. It does not. With ordinary
review costs it removes about 57%, and below roughly 38% accuracy it removes nothing at
all and starts adding work.
This is the most expensive error in AI business cases, because it is arithmetic on a
real measurement and it goes straight into a budget.
The refusal
Never multiply accuracy by volume or headcount. If the three cost constants have
not been measured, refuse to give a savings figure. Give the break-even instead, which
is computable from assumptions and honest about being one.
The three constants
Everything relative to doing the task from scratch (= 1.0):
|
what it is |
typical |
fromScratch |
doing the work with no draft |
1.0 |
reviewGoodDraft |
skim a correct draft, agree, move on |
0.3 to 0.4 |
reviewBadDraft |
read it, find the flaw, discard, start over |
1.3 to 1.6 |
reviewBadDraft is above 1.0 and that is the whole point. Handling a wrong answer costs
more than never having had one.
Procedure
- Time the three constants on real work. Twenty items each is usually enough to
separate 0.35 from 1.4. If you cannot measure them, say the numbers are assumed and
show the break-even curve rather than a point estimate.
breakEven(reviewGoodDraft, reviewBadDraft) from lib/stats.ts. Report it first.realSaving(accuracy, good, bad) for the actual figure.- Carry the accuracy interval through. A saving computed off a point estimate from
n=40 inherits that uncertainty. Report the saving at the lower accuracy bound too.
- Check the deployment shape. Draft-assist, threshold-gated automation and full
automation have different economics. A model that fails draft-assist can still pay
as a high-confidence filter over part of the volume.
- Name what the number excludes. Integration, monitoring, the reviewers' ramp,
escaped errors reaching a customer. These are usually larger than the modelling.
Output contract
- the three constants, measured or assumed, labelled which
- break-even accuracy
- real saving at the point estimate, and at the lower confidence bound
- the deployment shape assumed
- costs excluded from the figure
What this prevents
A business case built on "92% accurate, 40 people doing this, so 37 roles freed". The
defensible version is closer to 22, it depends on review costs nobody measured, and the
gap is discovered after the headcount plan is approved.
1---2name: automation-break-even3description: Convert model accuracy into an honest estimate of work saved, cost saved, or headcount impact. Use for AI business cases, ROI estimates, automation proposals, or any claim of the form "the model is X% accurate so it saves X% of the work". Measures the real review costs first and refuses to multiply.4---56# Accuracy is not savings78The model is 92% accurate, so it removes 92% of the work. It does not. With ordinary9review costs it removes about 57%, and below roughly 38% accuracy it removes nothing at10all and starts adding work.1112This is the most expensive error in AI business cases, because it is arithmetic on a13real measurement and it goes straight into a budget.1415## The refusal1617**Never multiply accuracy by volume or headcount.** If the three cost constants have18not been measured, refuse to give a savings figure. Give the break-even instead, which19is computable from assumptions and honest about being one.2021## The three constants2223Everything relative to doing the task from scratch (= 1.0):2425| | what it is | typical |26|---|---|---|27| `fromScratch` | doing the work with no draft | 1.0 |28| `reviewGoodDraft` | skim a correct draft, agree, move on | 0.3 to 0.4 |29| `reviewBadDraft` | read it, find the flaw, discard, start over | 1.3 to 1.6 |3031`reviewBadDraft` is above 1.0 and that is the whole point. Handling a wrong answer costs32more than never having had one.3334## Procedure35361. **Time the three constants on real work.** Twenty items each is usually enough to37 separate 0.35 from 1.4. If you cannot measure them, say the numbers are assumed and38 show the break-even curve rather than a point estimate.392. `breakEven(reviewGoodDraft, reviewBadDraft)` from `lib/stats.ts`. Report it first.403. `realSaving(accuracy, good, bad)` for the actual figure.414. **Carry the accuracy interval through.** A saving computed off a point estimate from42 n=40 inherits that uncertainty. Report the saving at the lower accuracy bound too.435. **Check the deployment shape.** Draft-assist, threshold-gated automation and full44 automation have different economics. A model that fails draft-assist can still pay45 as a high-confidence filter over part of the volume.466. **Name what the number excludes.** Integration, monitoring, the reviewers' ramp,47 escaped errors reaching a customer. These are usually larger than the modelling.4849## Output contract5051- the three constants, measured or assumed, labelled which52- break-even accuracy53- real saving at the point estimate, and at the lower confidence bound54- the deployment shape assumed55- costs excluded from the figure5657## What this prevents5859A business case built on "92% accurate, 40 people doing this, so 37 roles freed". The60defensible version is closer to 22, it depends on review costs nobody measured, and the61gap is discovered after the headcount plan is approved.
ityaadiii