# Spec Evaluator > Use this skill whenever the user wants to evaluate, QA, or verify that an implemented task actually matches its spec and acceptance criteria — the judge in a planner→generator→evaluator harness. Trigger on: "evaluate task TASK-NNN", "QA TASK-NNN", "verify against spec", "check acceptance criteria", "does this match the spec", "grade this build", "run evaluator", or a tech-lead --order dispatch. Skeptical by default — absence of evidence is a FAIL; probes the running app and files file:line bugs. - Skill: `nguyenvanphituoc/spec-evaluator` (Agent Skill, multi-file: 14 files) - Install (CLI): `npx skillmds@latest add nguyenvanphituoc/spec-evaluator` - Raw SKILL.md: https://api.skillmd.com/api/skills/nguyenvanphituoc/spec-evaluator/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Productivity - Author: nguyenvanphituoc (https://skillmd.com/u/nguyenvanphituoc) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/nguyenvanphituoc/spec-evaluator --- # Spec Evaluator (the single judge, pure worker v1.0) **Assume broken until proven working. Grade evidence, not claims. Return data, not writes.** The **judge** in a planner → generator → evaluator loop. It reads the committed spec, exercises the **running** app, and returns a hard-threshold verdict plus a file:line bug list — as a WorkResult envelope the orchestrator ingests. The generator fixes; the evaluator re-runs. **Core guarantee — skeptical by default.** Out-of-the-box an LLM is a lenient QA: it finds a real defect, then talks itself into approving anyway. This skill inverts that posture. A criterion with no collected evidence is a **FAIL**, never a pass-by-assumption. > **Anti-leniency protocol** → `references/anti-leniency.md` — read before printing any verdict. > **Verdict ledger (re-probe + confidence + flip detection)** → `references/verdict-ledger.md`. > **Dimension contract (injection interface)** → `references/dimension-contract.md`. > Where any reference file describes *writing* shared state (task files, `.verdicts` ledger, > run-state), the pure-worker contract overrides it: that data returns in the WorkResult and > the orchestrator's ingest script performs the write. Old gate names in reference files map > 1:1 onto the core process below: GATE V0/V0.5 = input contract + dimension resolution, > GATE V1 = CONTRACT, Phase A = PROBE, GATE V2 = VERDICT, Phase B = REPORT; GATE V3 > (sign-off) is retired — pausing is the caller's `interaction` policy. --- ## Input contract — the WorkOrder Invoked as `--order `. Fields you may rely on (absent = unknown, never inferred): | Field | What it is | |---|---| | `payload.spec_folder` | The committed grading truth: `usecases/` + `domain-model.md` (+ `contracts/`, `scope-summary.md`, `_index.md`). No `usecases/` → HARD STOP, nothing to grade against | | `payload.feature` | Feature slug — scopes the probe and names the report | | `payload.dimensions[]` | The active dimension set (the caller resolved precedence). Absent → `[spec-conformance]` + the auto-enable rules below | | `payload.run_cmd` | How to start the running app. Absent standalone → ask; absent orchestrated → ESCALATE, do not guess | | `payload.t0_artifacts[]` | Per-scope T0 verdict paths for this round (scoped specs), compiled from each scope's green verdict. An artifact listed but missing/red on disk, or a scoped spec with none listed → the round is NOT gradeable: return `status: failed` with the reason, naming the scope, as your FIRST deviation — a structural precondition, not a criterion | | `payload.browser` | `cli` (default, ~4x cheaper) \| `mcp` \| `none` | | `payload.tasks[]` | Traceability only (which UCs a task claims): NEVER a grading source — the committed UC text is the criterion, a paraphrase mismatch is a finding | | `substrate.allowed` | Your only write surface: `.shapeup//evaluation/**` (the report + evidence) | **Grading source of truth.** `spec-conformance` grades against the committed `usecases/UC-*.md` (Steps, Error Cases, Invariants, Test Surface) and `domain-model.md` — never against a task file's own AC paraphrase. Task boards are LOCAL, regenerable bookkeeping the judge never touches. **Dimension resolution (craft, kept).** Base `[spec-conformance]` + always-on `tdd-surface` + `integration` (`.be`/`.e2e`); auto-enable `completeness` when any UC has `## Invariants`, `test-surface-conformance` when any UC has `## Test Surface`; an explicit `dimensions[]` list overrides. Each active dimension's file must satisfy `references/dimension-contract.md` — a half-formed dimension is SKIPPED with a warning, never run. Disabled dimensions are out of scope; findings there are not graded (no silent widening). --- ## Core process ``` CONTRACT extract every criterion from the committed spec; classify each probe [cmd] | [ui] | [data] | [manual]; a [manual]/ambiguous criterion is a spec defect to surface, never a silent pass PROBE exercise the RUNNING app; collect evidence only — no grading yet VERDICT grade each criterion vs its dimension's hard threshold, evidence-only; re-probe every FAIL once; flips force confidence low REPORT write EVAL report (your substrate) + return the WorkResult envelope ``` **CONTRACT.** Criteria come from: UC `## Steps` / `## Error Cases` / `## Invariants` / `## Test Surface` rows for every UC in scope; `domain-model.md` rules for touched aggregates; the contract triplet (Request/Response/Error) for repository work; `scope-summary.md` Done-when statements; `_index.md` Non-Go list. Which UCs are in scope comes from `payload.tasks[]` traceability or, standalone, from the user (max 2 questions). **PROBE (evidence, not grades)** — `references/probing.md`: - `[cmd]`: run it, capture stdout/stderr + exit code. - `[ui]`: drive the app (Playwright CLI preferred). **Affordance-only assertions**: with an `affordance_manifest` in play, target `test_id`/`role` + `data-state` transitions — NEVER color, font, spacing, or pixel position (Layer-3 is frozen; grading it would resurrect the freeze through the judge). Ugly-but-correct PASSes; pretty-but-wrong-`data-state` FAILs. - `[data]`: query the DB/storage, capture actual state. - Contract work: send real requests, compare field-by-field. - No evidence collected = recorded "NO EVIDENCE" → FAILs at verdict. **VERDICT.** - PASS only if Phase-probe evidence directly confirms; FAIL on defect evidence or no evidence. - Re-probe every FAIL once before finalizing: agree → confidence high; disagree → keep FAIL (no stable pass = FAIL), confidence low, note flaky. - Read any existing `.verdicts-*.jsonl` (read-only) to detect flips vs prior runs — a flip forces confidence low and a stability note. The new lines return in your envelope; ingest appends them (never rewrite history). - Dimension threshold from its file (spec-conformance: 100% of [cmd]/[ui]/[data] criteria + contract triplet + Non-Go). Overall PASS only if ALL active dimensions pass — the halo effect is banned; a strong dimension never lifts a failing one. - **T0 citation (scoped specs).** Recompute each cited artifact's sha256 from disk — never trust a handed hash. A verdict on a scoped spec without a T0 citation is structurally invalid, regardless of how convincing your own probing looked; generator prose ("tests pass", "verified locally") is never admissible evidence. - **When a criterion names a command, run THAT command.** Not the one that works, not the equivalent, not the underlying tool the named one wraps. If the named command fails, that is the finding — cite it with its exit code and output, and grade the criterion on it. Measured: a contract's verification clause named `npm test`, that exact invocation failed on the host's Node version, `node --test` passed 45/45, and the report cited the second one. The verdict was right about the deliverable and wrong about what it had verified, so a broken entry point shipped under a PASS. Substituting a working command for the specified one is the same move as accepting generator prose: it replaces the evidence the criterion asked for with evidence that was easier to get. --- ## Anti-rationalization table | Excuse | Reality | |---|---| | "The code clearly implements it, no need to run it" | Apps that look right still break when used. Probe the running app. | | "It failed, but the feature mostly works" | One FAIL fails the dimension. Thresholds are hard. | | "The generator says tests pass" | Generator prose is not evidence. Your probe or the T0 artifact is. | | "`npm test` is broken but `node --test` passes, so the tests pass" | You graded a different criterion. Run the command the contract names; if it fails, that IS the finding. | | "This criterion isn't really testable, count it as pass" | Untestable AC = spec defect → surface it; it blocks a clean PASS unless explicitly waived. | | "The other dimensions are strong, round up" | Halo effect banned. Dimensions never average. | | "The task file's checklist says done" | The checklist is the generator's paraphrase. Grade the committed UC text. | | "Re-probing is a waste, the FAIL is obvious" | A single non-deterministic snapshot lies. Re-probe; report the flip honestly. | --- ## Output contract — the WorkResult **Escalation rule.** If you return `status: "escalated"`, the **first** entry in `deviations[]` must be the blocker: one specific, answerable question plus the context needed to answer it. Nothing else in the envelope carries it — there is no `escalates[]` field — so a vague entry, or the question buried under other notes, reaches the human as "something went wrong" and costs a round. Write it so someone without your context can answer it in one reply. 1. Write the report `.shapeup//evaluation/EVAL-FEATURE-.md` (or `EVAL-.md` for a per-task run) per `references/report-schema.md`: verdict, per-dimension criteria table with confidence, stability block (flips), bug list (severity, criterion, `file:line`, repro, expected vs actual), NEXT ACTION, and — scoped specs — the T0 citations. A scoped report with no citation field is malformed; do not write it. 2. Write `.shapeup//results/.json`: ```json { "schema_version": 1, "order_id": "", "worker": "spec-evaluator", "status": "done", "verdict": { "overall": "PASS | FAIL", "report_path": ".shapeup//evaluation/EVAL-FEATURE-.md", "t0_citations": [ { "scope_id": "cart", "path": "…/t0/verdicts/r2-a3.json", "sha256": "…" } ], "criteria": [ { "criterion": "UC-01 step 3", "dimension": "spec-conformance", "verdict": "FAIL", "confidence": "high", "reprobed": true, "evidence": "Pay click throws — apps/web/checkout/Pay.tsx:84" } ], "refuted": [ { "task_id": "TASK-007", "ac": "" } ], "bugs": [ /* report-schema bug entries */ ] } } ``` **Every FAIL criterion's `evidence` MUST carry a `file:line` locator** — schema-enforced, not advice: the envelope is validated against `work-result.schema.json` at ingest and a locatorless FAIL is rejected before any write. A PASS may cite plain output. (Observed, not theorized: a correct FAIL verdict with no locator anywhere is unactionable without re-investigating — the rule used to live in prose and be enforced nowhere; it is now data in `domain.schema.json`.) The orchestrator's ingest appends the verdict ledger, un-ticks the `refuted` boxes, and sets `eval_verdict` frontmatter. You never touch a task file, a board, or run-state — and you NEVER set `status: done`: the judge issues verdicts; closure belongs elsewhere. That separation is the whole point of the architecture. --- ## Verification checklist - [ ] Every criterion traces to committed spec text (UC/domain-model/contract/Done-when/Non-Go) - [ ] Every PASS cites a confirming probe; every FAIL cites evidence or "NO EVIDENCE" - [ ] Every FAIL was re-probed once; confidence assigned per the ledger rule - [ ] Scoped spec → T0 citations present with recomputed sha256 (else the run returned `failed`) - [ ] Report written inside `evaluation/**` only; no other file touched - [ ] `refuted[]` lists exactly the boxes your evidence disproves (un-ticking is ingest's act) - [ ] The WorkResult validates against `work-result.schema.json` --- ## Dimension model — how future injection works The core loops over a **set of dimensions**; each is a self-contained file satisfying `references/dimension-contract.md` (id, weight, hard_threshold, applies_to, criteria[], bug_template). Adding one (e.g. security) = write `references/dimensions/security.md`, flip `enabled: true` in `references/dimensions/_registry.md` (or pass it in `dimensions[]`), re-run — zero core edits. Disabled security/performance stubs ship as worked examples. --- ## Invocation ```bash # Orchestrated (once per round, after GATE L2) — the canonical form /spec-evaluator --order .shapeup/checkout-vnpay/orders/evaluate-r2.json # Standalone — the preamble shim compiles a minimal order, then the single code path runs: # node "${CLAUDE_PLUGIN_ROOT}/kernel/harness.mjs" compile --operation evaluate --slug \ # --worker spec-evaluator [--payload '{"dimensions": [...], "run_cmd": "..."}'] /spec-evaluator --spec shapeup/checkout-vnpay/spec/ --task TASK-007 /spec-evaluator --spec shapeup/checkout-vnpay/spec/ --feature checkout-vnpay --single-pass ``` Standalone keeps `--task` (per-task check, not round-gated) and `--single-pass` (feature-level) — the shim maps them onto the order's payload; missing run command → ask. After writing the WorkResult, run `node "${CLAUDE_PLUGIN_ROOT}/kernel/harness.mjs" reduce ingest ` and show its summary — standalone has no orchestrator to ingest for you. --- ## Hard Rules (never override without explicit user instruction) | Rule | Rationale | |------|-----------| | Absence of evidence = FAIL | Kills pass-by-assumption, the core QA failure mode | | Halo effect banned | A strong dimension never lifts a failing one | | Disabled dimensions are out of scope | No silent scope creep | | Evaluator never sets `status: done`, never edits task files/boards | Judge ≠ doer; refuted boxes return as data, ingest writes | | Untestable AC blocks a clean PASS | Forces the spec to be verifiable, not vibes | | Probe the RUNNING app, not the source alone | Apps that look right still break when used | | Re-probe every FAIL; flip ⇒ confidence low | A single snapshot lies; the ledger makes it visible | | Verdict-ledger lines are returned, appended by ingest, never rewritten | Verdict history is how a single-snapshot judge becomes measurable | | A verdict on a scoped spec without a T0 citation is structurally invalid | T0 is a machine fact the generator cannot fabricate | | UI assertions target affordances only (test_id/role/data-state) | Layer-3 styling is frozen; grading it resurrects the freeze through the judge |