kitfunso Eval-Driven Development
Evals are unit tests for AI development. Define success criteria BEFORE implementing, measure AFTER.
Two Eval Types
Capability Evals — Does the new thing work?
[CAPABILITY EVAL: feature-name]
Success Criteria:
- [ ] Criterion 1 (objectively verifiable)
- [ ] Criterion 2
- [ ] Criterion 3
Regression Evals — Did we break anything?
[REGRESSION EVAL: feature-name]
Existing Behavior:
- test-1: PASS/FAIL
- test-2: PASS/FAIL
Grader Types
| Type | When | How |
|---|---|---|
| Code-based | Deterministic outputs | grep patterns, run tests, build checks, file existence |
| Model-based | Open-ended outputs | Claude evaluates against rubric (use separate context) |
| Human-based | Subjective quality | Flag for manual review |
Metrics
- pass@k — "At least one success in k attempts." Target: pass@3 > 90%
- pass^k — "All k trials succeed." Higher bar for critical paths.
Workflow
- Define evals BEFORE coding (what does success look like?)
- Implement the feature
- Evaluate — run evals, collect pass rates
- Iterate — fix failures, re-evaluate
- Report — document final pass rates with the change
Key Insight
The reviewer/grader should be a SEPARATE context from the implementer. Author bias means the implementer will rate their own work too favorably. Use a subagent or separate pass for grading.