Agent Evals

Agent evals

"It seems better" is a sample of three, remembered favourably. An eval exists to turn a change in a prompt, model, tool or retrieval step into a number you can defend — and to tell you honestly how much of the system that number covers.

When this fires

Before a prompt, model, tool definition or retrieval change is called an improvement; when agent behaviour is depended on and must not silently regress; when a production failure needs to become something that cannot come back. It does not fire for deterministic code paths, which are tests.

Procedure

1. Name the decision the eval serves — "is this change an improvement", "can this ship", "which of these two prompts". A suite with no decision attached gets built once and read never. Write the decision above the suite.
2. Take cases from real traffic, not imagination. Pull them from logs, traces, support tickets and the failure that prompted this work. Invented cases test the behaviour you already thought of, which is the behaviour least likely to be broken. Freeze the set and version it with the code.
3. Stratify, and write the strata down. Happy path, ambiguous input, missing or empty data, long context, multi-step tool use, cases where the right answer is a refusal, and adversarial input. Record roughly how many cases sit in each. The strata you have no cases for are part of the result and get reported as uncovered.
4. Pin the expected outcome at the granularity that actually exists. Exact match only where there is one right answer. Otherwise a checkable property: parses against the schema, contains the account id it was given, calls the lookup tool before the write tool, stays under the token cap, refuses. A grader for "sounds good" measures nothing.
5. Grade with code wherever code can. Schema validation, regex and substring checks, tool-call sequence assertions, numeric tolerance, latency and token ceilings. Deterministic, cheap, and free of the judgment you are trying to measure.
6. If a model grades, the grader is itself under test. Give it explicit criteria rather than a quality adjective, label a held-out slice by hand, and report the grader's agreement with those labels. Pin the grader's model and prompt and version them with the suite — changing the grader silently changes every historical number. An uncalibrated grader produces a figure of unknown meaning, and saying so is better than quoting it.
7. Run each case more than once. Sampling is nondeterministic, so a single pass cannot separate a real change from noise. Report pass rate with the trial count, and run at the sampling settings the system actually ships with, not only at the most deterministic setting available.
8. Record the baseline before changing anything. Run the suite against the current production configuration and store the result with the configuration that produced it. A change with no recorded "before" is not measurable, only assertable.
9. Make one command run it. A suite that needs a manual setup step is skipped exactly when it matters. Keep it cheap enough to run on every change to the prompt.
10. Diff per case, in both directions. An unchanged aggregate routinely hides equal numbers of new passes and new failures. Report which specific cases flipped to passing and which flipped to failing; the second list is the finding.
11. Gate on the flips, not the average. Decide in advance which subset is blocking — safety, required refusals, output contract, anything with an external side effect. A regression there stops and asks rather than being absorbed into a better mean.
12. Run it where the change lands, pre-merge or pre-release. The eval informs a release decision; it does not make one. Deploying on a green suite is a separate, separately authorized action.
13. Feed every production failure back in as a case. The suite's value is the accumulated real failures it now refuses to let back through.

Checklist

• The decision this suite supports is written down
• Cases come from real traffic, including the failure that prompted the work
• Strata listed, with the uncovered ones named rather than omitted
• Every case has a checkable expectation, not an impression
• Deterministic graders used wherever a property can be checked in code
• Any model grader is calibrated against hand labels, and its agreement is reported
• Trials per case stated; pass rates carry their trial count
• Baseline recorded against a named configuration before the change
• Per-case flips reported in both directions
• Blocking subset defined, and a regression in it stops and asks

Failure handling

• Pass rate moved by a point or two — that is probably sampling. Increase trials or say the result is within noise. Do not narrate a rounding error as an improvement.
• The suite passes but production still fails — the case set is unrepresentative, not the production report. Add the failure as a case before touching anything else.
• A model grader disagrees with humans — fix the rubric or replace it with a code grader. Reporting the grader's number while knowing it disagrees is fabrication with a decimal point.
• Cases were written by looking at current outputs — they encode today's behaviour as correct and will pass forever. Rebuild them from the requirement or from real traffic.
• The suite is too slow or costly to run — shrink it to the blocking subset and say which cases are now only run on demand. An unrun suite is not a control.

Evidence to report

Name the suite: how many cases, where they came from, the strata and the uncovered ones. Then the numbers with their conditions — baseline and post-change pass rates, trials per case, the configuration and grader version behind each. Then the per-case flip lists, both directions. Quote the actual run output; a claimed pass rate is not a measured one.

Distinguish plainly what happened: the suite was written, executed against a named configuration, and produced a measured difference. A suite that has been written but not run proves nothing at all.

What a pass does not prove

A green suite says the measured cases behaved as expected under the sampled runs, on that configuration. It does not establish that unmeasured inputs are handled, that the system is safe against adversarial input it was never given, that latency or cost is acceptable unless graders measured them, or that anything is working in production. Refuse to report "evaluated" without the recorded baseline, the trial count and the flip lists — with any of those missing, the honest claim is that the suite ran, not that the change is an improvement.