TheophilusChinomona Eval Harness Skill
Purpose
Define, run, and track evaluations for features — the "unit tests of AI development."
Eval Types
Capability Evals
Test if Claude can do something:
[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
- [ ] Criterion 1
- [ ] Criterion 2
Expected Output: Description of expected result
Regression Evals
Ensure changes don't break existing functionality:
[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
- existing-test-1: PASS/FAIL
- existing-test-2: PASS/FAIL
Result: X/Y passed (previously Y/Y)
Metrics
pass@k
"At least one success in k attempts"
- pass@1: First attempt success rate
- pass@3: Success within 3 attempts
- Typical target: pass@3 > 90%
Workflow
1. Define (Before Coding)
/eval define feature-name
Creates .claude/evals/feature-name.md
2. Check (During Implementation)
/eval check feature-name
Runs current evals and reports status
3. Report (After Implementation)
/eval report feature-name
Generates full eval report
Storage
.claude/evals/
├── feature-xyz.md # Eval definition
├── feature-xyz.log # Eval run history
└── baseline.json # Regression baselines
Anti-Patterns
- Do NOT define evals after coding — define BEFORE
- Do NOT skip regression evals
- Do NOT use slow evals — keep them fast
- Do NOT fully automate security checks — human review required