Eval Driven Development

Eval-Driven Development

Coverage

The practice of building language-model-integrated systems by writing evaluations before and alongside the system, and using the eval suite's aggregated pass-rate signal to gate every change. Covers the statistical (not binary) nature of LLM evaluation, the five primitives (dataset, evaluation function, aggregation, iteration loop, regression budget), the judgment-mechanism taxonomy (programmatic / model-graded / human-graded / hybrid), the distinction between system-specific evals and canonical public benchmarks (MMLU, HumanEval, BIG-bench, GAIA, MT-Bench), why higher scores are not always improvements (Goodhart's Law), the difference between offline evals and production telemetry, and the eval-lifecycle archetypes (acceptance, regression, calibration, red-team, cross-model).

Philosophy

Building LLM-integrated systems without evals is shipping airplanes based on how good the model feels at the desk. The system's behavior is stochastic, the input space is open-ended, and the developer's pet examples are not a representative sample of what users will throw at it. An eval suite is the empirical measurement instrument that lets a team distinguish "the new prompt works better" from "the new prompt works better on the five examples I happened to try."

The discipline's hard part is not writing evals. It is choosing what to measure, encoding the choice into a grader the team agrees with, sampling a dataset that represents production, and resisting the gravitational pull of Goodhart's Law as the eval suite becomes the optimization target. Teams that get this right ship systems whose quality matches their team's stated definition of "good." Teams that get this wrong ship systems that ace evals and disappoint users.

Eval-driven development is not test-driven development with extra noise. It is empirical engineering applied to systems whose behavior is a distribution rather than a value. The vocabulary overlaps; the math underneath does not.

The Five Primitives In Practice

Judgment Mechanism Selection

A practical default: programmatic checks for the parts you can mechanically verify, model-graded for the open parts, periodic human review to calibrate.

Iteration Loop Discipline

The eval-driven iteration loop is the development cycle. Run the suite, diagnose failures, identify the change, re-run, gate the change on regression budget.

┌─────────┐    ┌──────────┐    ┌──────────────┐    ┌──────────┐
│ Eval    │ -> │ Diagnose │ -> │ Change       │ -> │ Re-run   │
│ baseline│    │ failures │    │ (prompt,     │    │ Eval     │
│         │    │          │    │  model,      │    │          │
│         │    │          │    │  retrieval,  │    │          │
│         │    │          │    │  tooling)    │    │          │
└─────────┘    └──────────┘    └──────────────┘    └──────────┘
                                                          │
                                                          v
                                                ┌──────────────────┐
                                                │ Compare against  │
                                                │ regression budget│
                                                │ Merge / iterate  │
                                                └──────────────────┘

The discipline is keeping the suite stable while the system changes. If both move in the same iteration, the comparison anchor is gone and the team is doing parallel experiments.

Public Benchmarks — Cited For Grounding, Not For Gating

Benchmarks measure cross-system capability against a shared standard. They predict how a model will do on the exact tasks the benchmark contains. They do not predict how your specific system, on your specific user inputs, with your specific prompts and retrieval, will perform.

The right use: pick a model partly on benchmark performance, then build system-specific evals to gate the actual deployment.

Goodhart's Law In Eval Practice

When the eval becomes the optimization target, the eval ceases to be a good measure. Symptoms:

• Pass-rate climbs while human reviewers' confidence in the system flattens or declines.
• Prompt changes produce phrasings the grader rewards but users dislike (e.g., verbose hedging that scores well on rubric, reads poorly on screen).
• The system memorizes patterns specific to the eval dataset (over-fitting to test cases).
• A held-out evaluation set, scored fresh, shows worse pass-rate than the development set.

Defenses:

• Hold out a portion of the dataset from active iteration. Score it periodically; if held-out and development pass-rates diverge, the iteration is over-fitting.
• Periodically refresh the eval dataset. Replace some examples with new production-sampled inputs to prevent the dataset from going stale.
• Calibrate model-graders against humans. A grader that has drifted from human judgment can produce high pass-rates on outputs humans dislike.
• Track multiple criteria. A single headline number is easier to over-fit than a panel of independent measures.

What This Skill Is Not

This skill is the concept of eval-driven development. Specific topics with their own scope:

• The mechanics of running evals in CI/CD pipelines belong to a tooling skill.
• The construction of individual eval rubrics, task sets, graders, and hard negatives belongs to agent-eval-design.
• The deterministic testing of non-LLM code belongs to testing-strategy.
• The production monitoring of running systems belongs to observability and reliability skills.
• The obra/superpowers test-driven-development skill (on skills.sh) is a process-shape workflow skill for general software TDD; this one is the concept-shape complement for the LLM-specific evaluation discipline.

Verification

After applying this skill, verify:

• An eval dataset exists, is checked into version control, and is representative of the system's actual production input distribution.
• Each eval criterion has a defined judgment mechanism (programmatic, model-graded, or human-graded), and the mechanism's known biases are accounted for.
• Aggregation reports pass-rate with sample size and either a confidence interval or a defined minimum-detectable-change threshold.
• Each eval has an explicit regression budget: gating (no regression allowed), optimizing (improvement gates merge), or watchful (tracked, not gated).
• Model-graded evals have been calibrated against human review on a sample within the past quarter (or whatever cadence the team has agreed to).
• A held-out portion of the dataset is reserved from active iteration and scored periodically to detect over-fitting.
• The eval suite is integrated into the change workflow — prompt changes, model upgrades, retrieval changes, and tooling changes are all gated by the suite.
• Public benchmarks (MMLU, HumanEval, etc.) are cited for model-selection grounding but are not the gating decision for system-specific quality.
• Production telemetry exists separately from the offline eval suite; one is not used as a substitute for the other.
• The shipping threshold is a product decision documented somewhere, not an emergent average across team opinion.

Do NOT Use When

Key Sources

• Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). "Measuring Massive Multitask Language Understanding". The MMLU benchmark paper; foundational reference for cross-system general-knowledge evaluation.
• Chen, M., Tworek, J., Jun, H., Yuan, Q., et al. (2021). "Evaluating Large Language Models Trained on Code". The HumanEval benchmark paper; foundational for code-generation evaluation.
• Srivastava, A., et al. (2022). "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models". The BIG-bench paper; canonical reference for breadth-of-capability evaluation across 200+ tasks.
• Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023). "GAIA: A Benchmark for General AI Assistants". The GAIA benchmark paper; canonical reference for evaluating multi-step assistant tasks with tool use.
• Zheng, L., Chiang, W.-L., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena". The MT-Bench paper; canonical reference for LLM-as-judge methodology, including known biases.
• OpenAI. Evals framework on GitHub. Open-source framework for writing and running LLM evals; documents the practical mechanics of the discipline.
• Anthropic. Building evals — Anthropic Cookbook and Evaluation guide. Practitioner-oriented guidance on building eval suites.
• UK AI Safety Institute. Inspect: An open-source evaluation framework. Open framework purpose-built for capability and safety evaluations of LLMs.
• Goodhart, C. (1975). "Problems of Monetary Management: The U.K. Experience." The origin of Goodhart's Law as commonly cited; "when a measure becomes a target, it ceases to be a good measure."
• Liang, P., et al. (2022). "Holistic Evaluation of Language Models". The HELM framework paper; argues for multi-metric eval across many dimensions as a counter to single-metric Goodharting.