AgentDB read-start has run. Check for existing eval definitions in _meta/research/.
Understand what behavior you're evaluating before writing evals.
- Spawn
agents/blind-evaluator.md as a fresh agent.
- Pass ONLY: problem statement, rubric (3-7 criteria with PASS conditions + weights), artifact path.
- Do NOT pass: implementer's checkpoint, summary, commit message, prompt, or expected solution.
- (gate: blind evaluator runs contamination check — if forbidden inputs detected, returns INVALID; clean inputs and retry)
- (gate: confidence < 0.7 from blind evaluator → escalate to human grader)
Two-phase eval protocol:
- Run 1: implementing agent solves cold, no eval feedback. Blind evaluator scores. This is the externally-reportable number.
- Run 2: implementing agent gets Run 1 score + rubric breakdown, then optimizes. For iteration only.
pass^k: "All k trials succeed"
- pass^3: 3 consecutive successes
- Use for critical paths (auth, payments)
See reference for calculation formula and worked examples.
See reference for full grader templates and examples.
Record eval type, pass rates, and any failures for future reference.
1---2name: eval3description: Eval-Driven Development (EDD) for AI workflows. pass@k metrics, capability evals, regression evals. Triggers: eval, edd, pass@k, capability, regression, benchmark.4---56<skill id="eval">78<prerequisite>9AgentDB read-start has run. Check for existing eval definitions in _meta/research/.10Understand what behavior you're evaluating before writing evals.11</prerequisite>1213<reference>14Skill-specific: skills/eval/reference/eval-research.md15</reference>1617<core_principles>181. DEFINE BEFORE CODE: Evals written first force clear thinking about success criteria.192. CODE GRADERS > MODEL GRADERS: Deterministic checks beat probabilistic judgments.203. STRUCTURAL SEPARATION FOR HIGH-STAKES: When stakes are real (security, payments, eval-of-evals, agent quality scoring), use the blind-evaluator agent — never self-score. Self-scoring inflates results ~36% structurally; procedural separation ("I won't peek") does not fix it.214. TRACK PASS@K: pass@1 (first attempt), pass@3 (within 3 attempts). Target pass@3 > 90%.225. REGRESSION BEFORE SHIP: Every change must pass existing evals before merge.236. FAST EVALS GET RUN: Slow evals get skipped. Keep evaluation fast.24</core_principles>2526<workflow>271. DEFINE: Write eval criteria before implementation. (gate: criteria exist in writing before any code)282. IMPLEMENT: Code to pass defined evals.293. EVALUATE: Run evals, record pass@k. (gate: pass@3 > 90% for capability; pass^3 = 100% for regression)304. REPORT: Document results in eval report format. See reference for template.31</workflow>3233<blind_evaluation_protocol>34Use when implementing agent would otherwise score its own output (high-stakes: security, payments, agent quality):35361. Spawn `agents/blind-evaluator.md` as a fresh agent.372. Pass ONLY: problem statement, rubric (3-7 criteria with PASS conditions + weights), artifact path.383. Do NOT pass: implementer's checkpoint, summary, commit message, prompt, or expected solution.394. (gate: blind evaluator runs contamination check — if forbidden inputs detected, returns INVALID; clean inputs and retry)405. (gate: confidence < 0.7 from blind evaluator → escalate to human grader)4142Two-phase eval protocol:43- Run 1: implementing agent solves cold, no eval feedback. Blind evaluator scores. This is the externally-reportable number.44- Run 2: implementing agent gets Run 1 score + rubric breakdown, then optimizes. For iteration only.45</blind_evaluation_protocol>4647<metrics>48pass@k: "At least one success in k attempts"49- pass@1: First attempt success rate50- pass@3: Success within 3 attempts (typical target: > 90%)5152pass^k: "All k trials succeed"53- pass^3: 3 consecutive successes54- Use for critical paths (auth, payments)5556See reference for calculation formula and worked examples.57</metrics>5859<grader_selection>601. Code-based (preferred): grep, test suite, build, type-check — deterministic, fast.612. Model-based: for open-ended outputs that can't be checked deterministically. Run multiple times, take majority.623. Human: required for security-sensitive changes, UX evaluation, legal/compliance.6364See reference for full grader templates and examples.65</grader_selection>6667<anti_patterns>68<block id="eval_after_code">Writing evals after implementation tests existing bugs, not requirements.</block>69<block id="model_grader_overuse">Model-based grading is slow and probabilistic. Prefer code graders.</block>70<block id="skip_regression">Every change must pass regression evals. No exceptions.</block>71<block id="slow_evals">Evals that take > 30s get skipped. Keep them fast.</block>72<block id="no_tracking">Track pass@k over time. Declining reliability is a signal.</block>73<block id="self_score">For any user-facing or high-stakes eval, the implementing agent scoring its own work inflates results ~36%. Spawn blind-evaluator instead.</block>74<block id="post_merge_eval">Evaluating against a codebase that already contains the canonical solution = answer key in the eval set. Use pre-merge snapshots or a separate fixture.</block>75<block id="greenfield_in_golden_dataset">Greenfield tickets in the golden eval set collapse to self=10, blind=3. Greenfields are not evaluable as solved tasks — exclude them from the dataset.</block>76<block id="context_breadth_before_baseline">Optimizing how much context the evaluator gets before establishing a baseline score = can't distinguish signal from noise. Run minimal-context baseline first, then test additions one at a time.</block>77</anti_patterns>7879<on_complete>80agentdb write-end '{"skill":"eval","eval_type":"capability|regression","pass_at_1":"<X%>","pass_at_3":"<Y%>","failures":["<list>"]}'8182Record eval type, pass rates, and any failures for future reference.83</on_complete>8485</skill>
hashgraph-online