Amey-Thakur LLM eval design
Without evals, every prompt edit is a blind bet and every model upgrade is a gamble. The eval suite is to LLM applications what the test suite is to code: built from real failures, run on every change, trusted because it is calibrated.
Method
- Harvest the eval set from production reality. Real (anonymized) inputs from logs (see llm-observability), especially the failures: user rephrases, thumbs-downs, support escalations, and the weird inputs nobody designed for; synthetic cases fill coverage gaps (edge lengths, adversarial phrasings, each language you support) but the spine is what users actually send (see ml-error-analysis: same loop, generative edition). Start small and honest: 50 well-chosen cases beat 5000 scraped ones.
- Grade with the cheapest sufficient judge. Exact/ programmatic checks wherever possible (structured output validates against schema: see structured-output; retrieval cites the right doc; the SQL runs and returns the expected rows); LLM-as-judge only for genuine open-ended quality, with a rubric per criterion (correctness, groundedness, tone: separate scores, not one vibe number).
- Calibrate the judge against humans. Label 50-100 cases yourself, measure judge-human agreement, iterate the rubric until agreement is strong on the dimensions that matter; recheck after judge-model upgrades. Known judge biases (position, verbosity, self-preference) need mitigations (randomize orderings, cap length credit): an uncalibrated judge is a random number generator with confidence (see model-evaluation's calibration ethic).
- Gate changes on eval runs. Every prompt edit, model swap, retrieval change, or parameter tune runs the suite before shipping (CI for prompts: see deployment-pipelines); compare against the incumbent with per-case diffs, not just aggregates: a +2% average hiding a regression on the refund-request cluster is a net loss (see model-evaluation's slicing). Version eval sets alongside prompts (see experiment-tracking).
- Test behaviors, not just answers. Invariance (paraphrase in, same answer out), refusal correctness (should-refuse and should-not-refuse sets: see llm-guardrails), format stability under adversarial input, and multi-turn behaviors for conversational systems (context carried, corrections honored: see conversation-design): single-turn QA evals miss the failures users actually hit.
- Close the loop continuously. New production failures enter the set weekly (the suite grows like a regression suite: every bug becomes a test: see regression-tests); sample live traffic for scheduled eval-in-production runs so drift (model provider updates, user-population shifts) surfaces on your dashboard, not in support tickets (see drift-monitoring's architecture aimed at quality).
Boundaries
- Offline evals estimate quality, not business impact; user-visible metrics (task completion, escalation rate) come from experiments (see ab-test-design).
- Judge scores are ordinal opinions, not measurements; never average across dimensions or compare across rubric versions, and re-baseline whenever the rubric changes.
- Public benchmarks measure the model, not your application; a model's leaderboard score transfers to your workload only if your eval set says so.