AI Evaluation
Use for model, prompt, retrieval, agent, or tool-workflow quality decisions. Do not rely on a few hand-picked examples.
Evaluation Design
- Define the user task, failure costs, quality dimensions, latency/cost constraints, and release decision.
- Build a versioned dataset from representative production-like cases, critical edge cases, adversarial inputs, and known regressions. Remove or protect sensitive data.
- Split data used for iteration from held-out evaluation where overfitting is plausible.
- Use deterministic graders for objective properties and calibrated rubrics or pairwise review for judgment-heavy quality.
- Track aggregate scores and important slices; preserve raw examples for error analysis.
- Compare against a frozen baseline and repeat stochastic runs enough to understand variance.
Use the evaluation-case schema for portable fixtures and the grader rubric checklist for judgment-heavy dimensions when the project has no established format.
Report dataset/version, configuration, grader limits, scores, confidence/variance, regressions, cost/latency, and release recommendation. Add confirmed failures to the regression set.
Cross-Model Benchmark Mode
Use this mode when the decision is which model or provider best performs a defined task. It compares models; it does not replace a representative evaluation design.
- Freeze the same prompt, tools, context, dataset, timeouts, and output contract for every candidate. Record unavoidable provider differences.
- Resolve currently available model identifiers from the provider or configured environment; do not rely on remembered names or silently substitute a model.
- Run representative repetitions when stochastic variance could change the ranking. Randomize output order before human or model judging when practical.
- Prefer deterministic graders for factual and structural requirements. For judgment-heavy quality, use a published rubric and blinded pairwise or independent scoring; never let a model grade its own output as the only judge.
- Track task success, important slice failures, latency distribution, token use, estimated cost, tool failures, and variance. A faster or cheaper model is not better when it misses release-critical cases.
- Show expected external cost before a paid benchmark and obtain approval when the run would create material spend. Never expose provider credentials in commands, artifacts, or reports.
- Save the exact configuration and raw sanitized outputs when the user requests a reproducible baseline. Compare future runs only when the task, dataset, grader, and environment remain compatible.
Recommend a model per workload or quality tier when the evidence supports segmentation; do not force a single global winner. State uncertainty and rerun criteria.
1---2name: ai-evaluation3description: Build reproducible evaluations for AI features using representative tasks, datasets, graders, baselines, slices, and release gates.4---56# AI Evaluation78Use for model, prompt, retrieval, agent, or tool-workflow quality decisions. Do not rely on a few hand-picked examples.910## Evaluation Design1112- Define the user task, failure costs, quality dimensions, latency/cost constraints, and release decision.13- Build a versioned dataset from representative production-like cases, critical edge cases, adversarial inputs, and known regressions. Remove or protect sensitive data.14- Split data used for iteration from held-out evaluation where overfitting is plausible.15- Use deterministic graders for objective properties and calibrated rubrics or pairwise review for judgment-heavy quality.16- Track aggregate scores and important slices; preserve raw examples for error analysis.17- Compare against a frozen baseline and repeat stochastic runs enough to understand variance.1819Use [the evaluation-case schema](assets/eval-case.schema.json) for portable fixtures and [the grader rubric checklist](references/grader-rubric.md) for judgment-heavy dimensions when the project has no established format.2021Report dataset/version, configuration, grader limits, scores, confidence/variance, regressions, cost/latency, and release recommendation. Add confirmed failures to the regression set.2223## Cross-Model Benchmark Mode2425Use this mode when the decision is which model or provider best performs a defined task. It compares models; it does not replace a representative evaluation design.2627- Freeze the same prompt, tools, context, dataset, timeouts, and output contract for every candidate. Record unavoidable provider differences.28- Resolve currently available model identifiers from the provider or configured environment; do not rely on remembered names or silently substitute a model.29- Run representative repetitions when stochastic variance could change the ranking. Randomize output order before human or model judging when practical.30- Prefer deterministic graders for factual and structural requirements. For judgment-heavy quality, use a published rubric and blinded pairwise or independent scoring; never let a model grade its own output as the only judge.31- Track task success, important slice failures, latency distribution, token use, estimated cost, tool failures, and variance. A faster or cheaper model is not better when it misses release-critical cases.32- Show expected external cost before a paid benchmark and obtain approval when the run would create material spend. Never expose provider credentials in commands, artifacts, or reports.33- Save the exact configuration and raw sanitized outputs when the user requests a reproducible baseline. Compare future runs only when the task, dataset, grader, and environment remain compatible.3435Recommend a model per workload or quality tier when the evidence supports segmentation; do not force a single global winner. State uncertainty and rerun criteria.
egeetas