Eval Engine
Spec in, a runnable verification layer out: gates, rubric, judge prompt, harness — with the one law enforced throughout: disqualifiers gate, tradeables score, never mixed.
Verification gates (defined first; output is blocked until all pass)
- G1 — Never mixed: every requirement lands as either a binary gate (disqualifier — fabrication, missing required elements, compliance misses) or a scored rubric dimension (tradeable — clarity, economy, tone). A disqualifier with a 1-5 score or a taste-call as a gate fails. Execution order stated in the deliverable: gates first, any failure = automatic fail, never averaged.
- G2 — Everything operational: every gate has a yes/no check procedure tagged
mechanical or judge; every rubric dimension has concrete 1- and 5-anchors; the judge prompt embeds them and demands quoted evidence; the harness instructions say who runs what, in what order, and what triggers recalibration.
- G3 — Spec-traced: every gate and dimension cites its spec phrase. Missing-but-obvious requirements are proposed as spec additions, never silently added.
Steps
- Harvest and sort. Split compound requirements; run the ship-it test on each (fails this, aces everything else — ship? No → gate; tradeable → rubric). Compliance-critical items ("must not miss X") always gate, with a seeded golden case as their check.
- Write the gates (3–6): name · binary check procedure · mechanical/judge tag · why-disqualifying. Vague disqualifiers get reformulated to checkable form ("no invented violations" → "every flagged rule exists verbatim in the policy doc").
- Write the rubric (3–6): name · concrete 1-anchor · 5-anchor · spec phrase. Anchors describe real output character, not adjectives.
- Build the judge prompt: paste-ready; embeds gates, anchors, and the required output shape (per-gate pass/fail with quoted evidence, per-dimension score with the sentence that earned it); forbids the judge from averaging gates or inventing criteria.
- Write the harness instructions: case format · gates-first run order · human spot-check policy (every gate failure + N random passes) · recalibration trigger (human-vs-judge gap ≥2 points → route to judge-calibration-auditor, rewrite the anchor not the verdict) · where new failures enter (failure-to-eval-capture).
- Gate pass. Re-run the sort check (G1), scan every artifact for operational completeness (G2), trace every row to spec (G3). Fix and re-run; maximum 2 repair loops, then report the failure.
Output format
EVAL: expense-report auditor (spec: 6 requirements → 3 gates · 3 dimensions)
GATES (binary — any failure = automatic fail; never averaged)
G1. No invented violations — every flagged rule exists verbatim in policy doc
[mechanical] — spec: "never invent a violation" — why: fabricated compliance flags are unshippable
G2. Rule citation present per flag [mechanical] — spec: "must cite the policy rule"
G3. Alcohol-on-client-dinner catch — seeded golden case must flag [mechanical] — spec: compliance-critical
RUBRIC (1-5, scored only when all gates pass)
R1. Plain-language clarity — 1: quotes policy verbatim with no translation · 5: one-sentence
plain explanation a submitter acts on — spec: "explain in plain language"
R2/R3. …
JUDGE PROMPT: [paste-ready block embedding the above, structured verdict + quoted evidence]
HARNESS: cases as {input, expected-gate-results}; run gates → rubric; human spot-check:
all gate-fails + 5 random passes/batch; recalibrate when human-judge gap ≥2 (fix the
anchor, not the verdict); new production failures → failure-to-eval-capture.
GATE CHECK: G1 pass (0 mixed) · G2 pass (all operational) · G3 pass (all spec-traced)
Hard rules
- Gates and scores never mix — the root eval failure this skill exists to prevent. When sorting is genuinely arguable, sort by stakes and flag the judgment.
- Every check must be runnable as written: a gate without a procedure or an anchor that fits any product gets rewritten, not shipped.
- Nothing enters the eval that isn't in the spec; gaps in the spec are proposed back, never patched silently.
- The judge is never the last word on its own drift — the harness always includes the human spot-check and the recalibration trigger.
Limitations
- The layer is designed, not executed — running cases, wiring CI, and collecting judge outputs is downstream work the harness instructions specify.
- Judge-tagged gates inherit judge fallibility; calibration (judge-calibration-auditor) is part of the system, not optional polish.
- Anchors calibrate; they don't eliminate drift — the ≥2-gap rule catches what they miss.
- A spec that's wrong produces a faithful eval of the wrong thing; the spec-trace makes that auditable, not impossible.
1---2name: eval-engine3description: Iterate-stage skill: turns a spec into the complete runnable verification layer — binary gates, anchored rubric, paste-ready judge prompt, and harness instructions — in one pass. Use when a feature needs its full eval built — 'create an eval for this spec', 'build the verification layer', 'spec to eval harness', 'turn this PRD into something we can run outputs through' — or when /pm routes such a request here. Do NOT use for the gates+rubric design artifact alone (prd-to-eval), for judge prompts over existing criteria (llm-as-judge-designer), for executing an eval over outputs, or for eval definitions.4---56# Eval Engine78Spec in, a runnable verification layer out: gates, rubric, judge prompt, harness — with the one law enforced throughout: disqualifiers gate, tradeables score, never mixed.910## Verification gates (defined first; output is blocked until all pass)1112- **G1 — Never mixed:** every requirement lands as either a binary gate (disqualifier — fabrication, missing required elements, compliance misses) or a scored rubric dimension (tradeable — clarity, economy, tone). A disqualifier with a 1-5 score or a taste-call as a gate fails. Execution order stated in the deliverable: gates first, any failure = automatic fail, never averaged.13- **G2 — Everything operational:** every gate has a yes/no check procedure tagged `mechanical` or `judge`; every rubric dimension has concrete 1- and 5-anchors; the judge prompt embeds them and demands quoted evidence; the harness instructions say who runs what, in what order, and what triggers recalibration.14- **G3 — Spec-traced:** every gate and dimension cites its spec phrase. Missing-but-obvious requirements are proposed as spec additions, never silently added.1516## Steps17181. **Harvest and sort.** Split compound requirements; run the ship-it test on each (*fails this, aces everything else — ship?* No → gate; tradeable → rubric). Compliance-critical items ("must not miss X") always gate, with a seeded golden case as their check.192. **Write the gates (3–6):** name · binary check procedure · mechanical/judge tag · why-disqualifying. Vague disqualifiers get reformulated to checkable form ("no invented violations" → "every flagged rule exists verbatim in the policy doc").203. **Write the rubric (3–6):** name · concrete 1-anchor · 5-anchor · spec phrase. Anchors describe real output character, not adjectives.214. **Build the judge prompt:** paste-ready; embeds gates, anchors, and the required output shape (per-gate pass/fail with quoted evidence, per-dimension score with the sentence that earned it); forbids the judge from averaging gates or inventing criteria.225. **Write the harness instructions:** case format · gates-first run order · human spot-check policy (every gate failure + N random passes) · recalibration trigger (human-vs-judge gap ≥2 points → route to judge-calibration-auditor, rewrite the anchor not the verdict) · where new failures enter (failure-to-eval-capture).236. **Gate pass.** Re-run the sort check (G1), scan every artifact for operational completeness (G2), trace every row to spec (G3). Fix and re-run; maximum 2 repair loops, then report the failure.2425## Output format2627```28EVAL: expense-report auditor (spec: 6 requirements → 3 gates · 3 dimensions)29GATES (binary — any failure = automatic fail; never averaged)30G1. No invented violations — every flagged rule exists verbatim in policy doc31 [mechanical] — spec: "never invent a violation" — why: fabricated compliance flags are unshippable32G2. Rule citation present per flag [mechanical] — spec: "must cite the policy rule"33G3. Alcohol-on-client-dinner catch — seeded golden case must flag [mechanical] — spec: compliance-critical34RUBRIC (1-5, scored only when all gates pass)35R1. Plain-language clarity — 1: quotes policy verbatim with no translation · 5: one-sentence36 plain explanation a submitter acts on — spec: "explain in plain language"37R2/R3. …38JUDGE PROMPT: [paste-ready block embedding the above, structured verdict + quoted evidence]39HARNESS: cases as {input, expected-gate-results}; run gates → rubric; human spot-check:40all gate-fails + 5 random passes/batch; recalibrate when human-judge gap ≥2 (fix the41anchor, not the verdict); new production failures → failure-to-eval-capture.42GATE CHECK: G1 pass (0 mixed) · G2 pass (all operational) · G3 pass (all spec-traced)43```4445## Hard rules46471. Gates and scores never mix — the root eval failure this skill exists to prevent. When sorting is genuinely arguable, sort by stakes and flag the judgment.482. Every check must be runnable as written: a gate without a procedure or an anchor that fits any product gets rewritten, not shipped.493. Nothing enters the eval that isn't in the spec; gaps in the spec are proposed back, never patched silently.504. The judge is never the last word on its own drift — the harness always includes the human spot-check and the recalibration trigger.5152## Limitations5354- The layer is designed, not executed — running cases, wiring CI, and collecting judge outputs is downstream work the harness instructions specify.55- Judge-tagged gates inherit judge fallibility; calibration (judge-calibration-auditor) is part of the system, not optional polish.56- Anchors calibrate; they don't eliminate drift — the ≥2-gap rule catches what they miss.57- A spec that's wrong produces a faithful eval of the wrong thing; the spec-trace makes that auditable, not impossible.
Abhillashjadhav