# Eval Set Builder

> Build an evaluation set from real traffic and check whether it is large enough to support the error budget being claimed. Use when someone wants to measure model accuracy, certify an error rate, decide if a model is safe to deploy, or asks "how many test cases do we need". Refuses to certify a budget the sample cannot support.

- Skill: `ityaadiii/eval-set-builder` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ityaadiii/eval-set-builder`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ityaadiii/eval-set-builder/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: ityaadiii (https://skillmd.com/u/ityaadiii)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ityaadiii/eval-set-builder

---


# Building an eval set that can actually answer the question

Most eval sets are assembled to a round number. Someone picks 50 or 100 items because
those feel like enough. Then a 2% error budget gets claimed off a sample that could
never have detected a 2% error rate in the first place.

## The refusal

**If the sample is too small to certify the budget, say so and stop.** Do not report a
pass. "No errors found in 60 items" is a fact about the sample, not about the model.

Rule of three: with zero observed errors in n items, the 95% upper bound on the true
error rate is roughly 3/n. Certifying "under 2%" therefore needs about 150 clean items.
Under 1% needs 300.

```bash
node lib/stats.ts   # see certifies() and minItemsToCertify()
```

Run `certifies(errors, n, budget)` and report `upperBound`, not the point estimate.

## Procedure

1. **Get the budget first.** Ask what error rate the process can absorb before anyone
   picks a sample size. If nobody can answer, that is the finding. Stop there.
2. **Compute the floor.** `minItemsToCertify(budget)`. State it before any data is
   collected, so the number is not negotiated after the fact.
3. **Sample from real traffic, not from imagination.** Generated or hand-written items
   are easier than production and will overstate accuracy. If only synthetic data is
   available, label the whole result as non-evidential.
4. **Stratify by the slices that matter** (channel, language, document type, region).
   An aggregate that hides a collapsed slice is worse than no number.
5. **Check for contamination.** If items could plausibly be in a model's training data,
   say so. Public benchmarks almost always are.
6. **Get labels you can defend.** If a field is a judgement call, it needs two
   annotators and an agreement statistic before it goes in a composite score. Grade it,
   show it, exclude it from the headline until then.

## Output contract

Report every one of these, in this order:

- error budget, and where it came from
- items needed to certify it, items actually collected
- observed errors, and the Wilson **upper** bound
- `certified: true | false`
- slices, with per-slice counts, flagging any slice too small to speak for itself
- fields excluded from the composite, and why

## What this prevents

A model gets deployed on "99% accurate, no errors in our test set", where the test set
had 40 items. The honest version of that sentence is "our error rate is somewhere
under 7%, and we cannot see anything finer than that yet."