Building an eval set that can actually answer the question
Most eval sets are assembled to a round number. Someone picks 50 or 100 items because
those feel like enough. Then a 2% error budget gets claimed off a sample that could
never have detected a 2% error rate in the first place.
The refusal
If the sample is too small to certify the budget, say so and stop. Do not report a
pass. "No errors found in 60 items" is a fact about the sample, not about the model.
Rule of three: with zero observed errors in n items, the 95% upper bound on the true
error rate is roughly 3/n. Certifying "under 2%" therefore needs about 150 clean items.
Under 1% needs 300.
node lib/stats.ts # see certifies() and minItemsToCertify()
Run certifies(errors, n, budget) and report upperBound, not the point estimate.
Procedure
- Get the budget first. Ask what error rate the process can absorb before anyone
picks a sample size. If nobody can answer, that is the finding. Stop there.
- Compute the floor.
minItemsToCertify(budget). State it before any data is
collected, so the number is not negotiated after the fact.
- Sample from real traffic, not from imagination. Generated or hand-written items
are easier than production and will overstate accuracy. If only synthetic data is
available, label the whole result as non-evidential.
- Stratify by the slices that matter (channel, language, document type, region).
An aggregate that hides a collapsed slice is worse than no number.
- Check for contamination. If items could plausibly be in a model's training data,
say so. Public benchmarks almost always are.
- Get labels you can defend. If a field is a judgement call, it needs two
annotators and an agreement statistic before it goes in a composite score. Grade it,
show it, exclude it from the headline until then.
Output contract
Report every one of these, in this order:
- error budget, and where it came from
- items needed to certify it, items actually collected
- observed errors, and the Wilson upper bound
certified: true | false- slices, with per-slice counts, flagging any slice too small to speak for itself
- fields excluded from the composite, and why
What this prevents
A model gets deployed on "99% accurate, no errors in our test set", where the test set
had 40 items. The honest version of that sentence is "our error rate is somewhere
under 7%, and we cannot see anything finer than that yet."
1---2name: eval-set-builder3description: Build an evaluation set from real traffic and check whether it is large enough to support the error budget being claimed. Use when someone wants to measure model accuracy, certify an error rate, decide if a model is safe to deploy, or asks "how many test cases do we need". Refuses to certify a budget the sample cannot support.4---56# Building an eval set that can actually answer the question78Most eval sets are assembled to a round number. Someone picks 50 or 100 items because9those feel like enough. Then a 2% error budget gets claimed off a sample that could10never have detected a 2% error rate in the first place.1112## The refusal1314**If the sample is too small to certify the budget, say so and stop.** Do not report a15pass. "No errors found in 60 items" is a fact about the sample, not about the model.1617Rule of three: with zero observed errors in n items, the 95% upper bound on the true18error rate is roughly 3/n. Certifying "under 2%" therefore needs about 150 clean items.19Under 1% needs 300.2021```bash22node lib/stats.ts # see certifies() and minItemsToCertify()23```2425Run `certifies(errors, n, budget)` and report `upperBound`, not the point estimate.2627## Procedure28291. **Get the budget first.** Ask what error rate the process can absorb before anyone30 picks a sample size. If nobody can answer, that is the finding. Stop there.312. **Compute the floor.** `minItemsToCertify(budget)`. State it before any data is32 collected, so the number is not negotiated after the fact.333. **Sample from real traffic, not from imagination.** Generated or hand-written items34 are easier than production and will overstate accuracy. If only synthetic data is35 available, label the whole result as non-evidential.364. **Stratify by the slices that matter** (channel, language, document type, region).37 An aggregate that hides a collapsed slice is worse than no number.385. **Check for contamination.** If items could plausibly be in a model's training data,39 say so. Public benchmarks almost always are.406. **Get labels you can defend.** If a field is a judgement call, it needs two41 annotators and an agreement statistic before it goes in a composite score. Grade it,42 show it, exclude it from the headline until then.4344## Output contract4546Report every one of these, in this order:4748- error budget, and where it came from49- items needed to certify it, items actually collected50- observed errors, and the Wilson **upper** bound51- `certified: true | false`52- slices, with per-slice counts, flagging any slice too small to speak for itself53- fields excluded from the composite, and why5455## What this prevents5657A model gets deployed on "99% accurate, no errors in our test set", where the test set58had 40 items. The honest version of that sentence is "our error rate is somewhere59under 7%, and we cannot see anything finer than that yet."
ityaadiii