Matrixx0070 When to use
Use this when an LLM feature is real enough that a silent quality regression would hurt users, and "it looked fine when I tried it" is no longer acceptable proof.
Not for: the first prompt sketch (iterate by hand first, see ai-prompt-design), or pure latency/cost tuning (see ai-cost-latency).
Method
- Collect a dataset from real traffic and known failures, not invented cases. Aim for 50-200 items; label each with the expected outcome or a rubric.
- Split into a dev set (you tune against) and a frozen holdout (you only measure). Never tune on the holdout.
- Pick a grader per case type. Decision point: exact/enum/JSON → programmatic assertion; open-ended → LLM-as-judge with a rubric; factual → check against a reference. Prefer the cheapest grader that is trustworthy.
- Validate the LLM judge itself: have it grade 20 human-labeled items; if agreement is low, tighten the rubric before trusting it.
- Compute aggregate metrics (accuracy, pass rate, per-category breakdown) plus a list of individual failures for inspection.
- Set a regression gate: a threshold that must hold in CI before a prompt/model change merges. Decision point: block on holdout pass-rate drop OR any critical-category failure.
- Re-run on every prompt, model, or retrieval change; archive scores per version.
Example
Feature: invoice field extraction. Dataset: 120 real invoices, expected JSON per invoice.
grader = exact_match(pred["total"], gold["total"])
and iso_date(pred["due_date"]) == gold["due_date"]
Baseline (prompt v3): 108/120. Candidate (v4, added a few-shot example): 116/120, but 2 NEW failures in the "foreign currency" category. Gate rule "no category may regress" blocks the merge until v5 fixes currency, reaching 119/120.
Pitfalls
- Vibes as eval. Eyeballing a few outputs. If it is not scored on a frozen set, it is not an eval.
- Trusting an unvalidated judge. LLM judges have biases (length, position). Calibrate against human labels first.
- Aggregate blindness. A steady average hides a tanked subcategory. Always break down by category.
- Leaky holdout. Tuning until the holdout passes turns it into a training set. Touch it only to measure.
- Non-determinism ignored. One run at temperature>0 is a coin flip, not a score. Run each item 3-5 times (or pin temperature=0 where valid) and report mean ± spread; a 2-point delta inside the noise band is not a regression or a win.
- Sample size too small to gate on. With n=20, an 85%→80% "drop" is one flipped item. State n next to every score; don't hard-gate categories with fewer than ~20 items — flag them for human review instead.
Verifying the eval itself
Before trusting any verdict, confirm the harness did what you asked — tools and CI runners silently drop parameters:
- Check the actual run count and item count in the output against what you requested.
- Check that every custom assertion appears in the per-item results — an assertion that no item ever fails (or that both sides always fail) is non-discriminating and must be rewritten or dropped.
- Spot-read 3-5 individual gradings per run; a judge that gives identical scores to visibly different outputs is broken.
Output format
# Eval Suite: <feature>
DATASET: <n> items | source | dev/holdout split
LABELS: <expected outcome or rubric>
GRADERS:
- <case type> -> <programmatic | llm-judge | reference>
JUDGE CALIBRATION: agreement=<x> on <n> human labels
METRICS: overall=<x> | per-category={...}
GATE: block merge if holdout<<t> OR any category regresses
RESULTS: v<n> <score> | failures=[...]