Evaluating AI output
Code is deterministic; LLM output isn't. A feature that "worked once" can fail the next call on the same input. Evals are the unit tests of AI work — they measure how often and how well the output meets the bar, not just that it can.
Read first (always)
List learnings/ and read anything relevant — provider quirks, rubric calibration, and known-flaky cases for this project belong there.
Define expected behavior BEFORE you implement
Write the eval first: the inputs, what a good output looks like, and what must never happen. If you can't state how you'd grade it, you don't yet understand the feature.
Measure across repeated trials
Run each case k times (LLM output varies) and report:
- pass@k — at least one of k attempts succeeds. Measures capability ("can it do this at all?"). Typical target: pass@3 > 90%.
- pass^k — all k attempts succeed. Measures stability/reliability ("does it do this every time?"). Use for critical paths (auth, money, irreversible actions). pass^3 means 3 consecutive clean runs.
A feature can have high pass@k but low pass^k — impressive once, unreliable in production. Match the metric to the risk.
Three graders (use the cheapest that's trustworthy)
| Grader |
How |
Use for |
| Code-based |
Deterministic check — regex/grep, schema/JSON validation, an assertion, a tool call that must appear |
Anything machine-verifiable (format, presence, exact values). Always prefer this. |
| Model-based |
A model scores the output 1–5 against a written rubric |
Quality/judgment that code can't check (relevance, tone, reasoning). Calibrate the rubric on a few human-labeled examples. |
| Human |
A person reviews, tagged risk LOW / MED / HIGH |
High-stakes or ambiguous cases. Never fully automate security review — keep a human in the loop there. |
Build the eval set
Cover representative cases, adversarial/edge cases (the failure modes from your threat model — see skills/threat-modeling), and regression cases (every bug becomes a permanent eval). Keep a baseline (committed scores) and gate changes on "no regression vs. baseline". Store the eval set, the baseline, and run logs alongside the feature.
Boundary with the testing skills
- Deterministic logic (parsing, math, control flow around the model) →
skills/writing-tests / developing-features-tdd. Don't write an eval for what a unit test can assert.
- Non-deterministic output quality (does the model's answer meet the bar, reliably) → here.
- For AI runtime resilience (timeouts, fallbacks, circuit breakers, kill switch), see
developing-features-tdd/references/ai-runtime.md.
Capture a learning
If you find a rubric that calibrates well, a recurring failure class, or a provider quirk, append a learnings/YYYY-MM-DD-slug.md (or use /learn).
See also
skills/writing-tests, skills/developing-features-tdd — deterministic coverage.skills/threat-modeling — abuse cases feed the adversarial eval set.skills/building-langflow-components — when the AI feature is a Langflow component.
1---2name: evaluating-ai-output3description: Evaluate non-deterministic LLM/AI output with evals instead of one-shot "it worked" — define expected behavior first, measure pass@k / pass^k, and grade with code / model / human graders. Use when building or changing an AI/LLM feature, an agent, a prompt, a RAG pipeline, or a classifier, where a single good run is not proof of correctness. Complements writing-tests (deterministic logic) and developing-features-tdd.4license: MIT5---67# Evaluating AI output89Code is deterministic; LLM output isn't. A feature that "worked once" can fail the next call on the same input. **Evals are the unit tests of AI work** — they measure how *often* and how *well* the output meets the bar, not just that it can.1011## Read first (always)1213List `learnings/` and read anything relevant — provider quirks, rubric calibration, and known-flaky cases for this project belong there.1415## Define expected behavior BEFORE you implement1617Write the eval first: the inputs, what a good output looks like, and what must never happen. If you can't state how you'd grade it, you don't yet understand the feature.1819## Measure across repeated trials2021Run each case **k times** (LLM output varies) and report:2223- **pass@k** — *at least one* of k attempts succeeds. Measures **capability** ("can it do this at all?"). Typical target: pass@3 > 90%.24- **pass^k** — *all* k attempts succeed. Measures **stability/reliability** ("does it do this every time?"). Use for **critical paths** (auth, money, irreversible actions). pass^3 means 3 consecutive clean runs.2526A feature can have high pass@k but low pass^k — impressive once, unreliable in production. Match the metric to the risk.2728## Three graders (use the cheapest that's trustworthy)2930| Grader | How | Use for |31|---|---|---|32| **Code-based** | Deterministic check — regex/`grep`, schema/JSON validation, an assertion, a tool call that must appear | Anything machine-verifiable (format, presence, exact values). Always prefer this. |33| **Model-based** | A model scores the output 1–5 against a written rubric | Quality/judgment that code can't check (relevance, tone, reasoning). Calibrate the rubric on a few human-labeled examples. |34| **Human** | A person reviews, tagged risk LOW / MED / HIGH | High-stakes or ambiguous cases. **Never fully automate security review** — keep a human in the loop there. |3536## Build the eval set3738Cover **representative** cases, **adversarial/edge** cases (the failure modes from your threat model — see `skills/threat-modeling`), and **regression** cases (every bug becomes a permanent eval). Keep a **baseline** (committed scores) and gate changes on "no regression vs. baseline". Store the eval set, the baseline, and run logs alongside the feature.3940## Boundary with the testing skills4142- **Deterministic logic** (parsing, math, control flow around the model) → `skills/writing-tests` / `developing-features-tdd`. Don't write an eval for what a unit test can assert.43- **Non-deterministic output quality** (does the model's answer meet the bar, reliably) → here.44- For AI runtime resilience (timeouts, fallbacks, circuit breakers, kill switch), see `developing-features-tdd/references/ai-runtime.md`.4546## Capture a learning4748If you find a rubric that calibrates well, a recurring failure class, or a provider quirk, append a `learnings/YYYY-MM-DD-slug.md` (or use `/learn`).4950## See also5152- `skills/writing-tests`, `skills/developing-features-tdd` — deterministic coverage.53- `skills/threat-modeling` — abuse cases feed the adversarial eval set.54- `skills/building-langflow-components` — when the AI feature is a Langflow component.
Cristhianzl