Build Evals

build-evals — the eval builder

!cat .forge/state.json 2>/dev/null || echo NO_STATE

Generate the complete evals/ tree for a target plugin. In new mode the plugin does not exist yet: tasks derive from design/CONTRACT.md (the PDR), not from code. The suite this skill emits becomes the definition of "done" for the entire build — treat every task as a contract clause, not a test.

This skill drafts and lints the suite. It never arms, freezes, or scores it — those are /plugin-forge:arm-evals and bin/forge-eval. It is model-invocable by design (decision D2): the forge conductor chains it via the Skill tool; safety comes from the state guard below, not from invocation control.

State guard (new mode only)

Apply this table to the injected state line above before doing anything else in new mode (also the default when no argument is given). retrofit and mine skip this guard entirely — they run standalone against any plugin, with or without a forge pipeline (see their sections).

Mode: new (pipeline phase)

Work through the steps in order. Load the referenced file before executing its step — each reference is binding, not advisory.

1. Map the surface from the contract

Read design/CONTRACT.md. Extract, per capability row:

• the chosen composition (skills, hooks, agents, MCP servers, scripts) and its archetype
• the C5 Testability column: grader type, observable signal, and planned task IDs — this column IS the task backlog; build-evals exists to make those IDs real
• the C1 Trigger column: every auto-invoked component (model-invoked skill, hook, agent with proactive description) needs positive AND negative trigger tasks
• external dependencies (MCP servers, APIs, corpora) and whether the contract chose live or mock for each

Also read design/INTENT.md for the runtime-environment answers (headless/cloud/local) — they constrain what graders can observe.

2. Interview — propose clusters, recommend, WAIT

Never one-shot the suite (interviewing beats one-shot generation — LangChain's central empirical finding). Use AskUserQuestion to:

1. Propose 3–5 testable ability clusters derived from the PDR rows, each with a one-line description, its archetype, and a rough task count. Recommend one ordering and say why. WAIT for the user's choice before generating anything.
2. For every external dependency, ask live-vs-mock. Default recommendation: mock fixture stubs for write-heavy, paid, or rate-limited tools; live only for cheap read-only calls the user explicitly accepts. Record the decision per dependency — the harness stores which mode each result came from.

For scientific or business domains where fixture corpora need domain-real content, offer to run the /plugin-forge:skill-forge research pipeline to ground fixtures in real sources instead of inventing plausible-looking data (decision D10).

3. Scaffold

Copy the scaffold from ${CLAUDE_PLUGIN_ROOT}/templates/evals/ into the target plugin as evals/, and create the sibling evals-holdout/ next to it (NOT inside evals/). The scaffold provides registry.yaml, targets/{default,no-plugin,crowded}.yaml, suites/{capability,regression,triggers}.yaml, judges/{fact,rubric,battle}.yaml, labels/labels.jsonl, and tasks/_example/. Delete tasks/_example/ after real tasks exist.

Then vendor the harness runner (stdlib-only, so the suite outlives plugin-forge):

mkdir -p <plugin_dir>/evals/bin && cp "${CLAUDE_PLUGIN_ROOT}/scripts/harness/run.py" <plugin_dir>/evals/bin/run.py && chmod +x <plugin_dir>/evals/bin/run.py

4. Author tasks

Follow references/task-authoring.md for everything about tasks: the exact task.yaml schema, the 20–50 task budget, the two-experts bar, the REQUIRED reference solution, the goal-condition grammar for instruction.md, fixture rules, the holdout split, and elicitation_answers. Every task ID planned in the contract's C5 column must exist in evals/tasks/ or evals-holdout/tasks/ when this step ends.

5. Choose graders

Follow references/grader-menu.md: deterministic-first hierarchy (workspace → transcript → state_check → judge → trigger, plus the simulate_user driver), archetype routing, the mandatory numeric tolerance, the never-assert-tool-ORDER rule, and the crowded-listing requirement for trigger tasks.

6. Author judges

Only where the grader menu genuinely runs out of deterministic options. Follow references/judge-authoring.md: modelgraded-spec YAML shape, one rubric dimension per isolated judge call, Unknown always legal, pinned cheap judge model different from the target model, anti-echo phrasing. Seed evals/labels/labels.jsonl with the unlabeled sample skeleton — calibration itself is arm-evals' gate, not this skill's.

7. Generate the elicitation auto-answer hook (when applicable)

If the target plugin bundles or drives an MCP server that uses elicitation, headless eval runs will hang or die — no dialog exists under claude -p. For every such server:

• set elicitation_answers in each task.yaml that exercises it (see task-authoring.md)
• emit evals/hooks/elicitation-auto-answer.sh (bash, set -euo pipefail, executable) and evals/hooks/elicitation-settings.json registering it as an exec-form Elicitation hook: "command": "${CLAUDE_PLUGIN_ROOT}/evals/hooks/elicitation-auto-answer.sh" — harness runs merge this settings file; production users never load it. The script reads the elicitation request JSON from stdin, looks up the requested fields in .forge-eval/elicitation-answers.json (staged into the workspace by the harness from the task's elicitation_answers), and prints the matching answers object to stdout with exit 0. No match → one-line stderr message and exit 2, so the trial fails loudly instead of hanging.

8. Lint the suite

Run every mechanical lint in references/antipatterns.md (L01–L16) and fix all findings before proceeding. L01 (numeric tolerance) and L02 (task-text/grader consistency) are the builder meta-eval contract: plugin-forge's own dogfood suite feeds this skill adversarial specs engineered to reproduce the CORE-Bench and METR grader bugs, and passes only if the emitted suite provably cannot contain them. Do not hand a suite to arm-evals with any lint red.

9. Smoke-run drafts and hand off

An eval that has never run is not delivered:

• Run forge-eval reference --task <ID> for at least 3 representative tasks (one per archetype used) and show the user the graded output. Use forge-eval run --dry-run for a cost estimate of the full suite; state it plainly — trials × tasks × judges is real money, and --max-cost-usd exists for a reason.
• Present the suite summary: task count per suite (capability/regression/triggers), holdout count, judge dimensions, live-vs-mock map, estimated full-run cost.
• Tell the user the exact next command: /plugin-forge:arm-evals — the four arming gates (reference-green, red baseline, judge calibration, freeze) all run there.

Do not modify .forge/state.json. The phase advances to armed only when arm-evals freezes the suite.

Mode: retrofit

Standalone — no forge pipeline, no state guard, works on any installed or in-repo plugin.

1. Map the plugin surface from files (there is no PDR): every skills/*/SKILL.md and command frontmatter (descriptions = trigger contracts), agents/*.md, hooks/hooks.json (matchers = firing contracts), .claude-plugin/plugin.json, .mcp.json. Derive one ability cluster per user-facing capability and one trigger axis per auto-invoked component. MCP tool checks must use plugin-scoped names: mcp__plugin_<plugin>_<server>__<tool>.
2. Then run steps 2–9 above unchanged, writing evals/ and evals-holdout/ into the plugin's own directory. Where step 1 of new mode says CONTRACT.md, use the surface map; flag any capability whose observable signal is unclear as "unverifiable — needs a design decision" instead of inventing a grader for it.
3. Recommend arming: retrofit suites get the same four gates via /plugin-forge:arm-evals <plugin-path> before their scores mean anything.

Mode: mine

Standalone — converts captured production failures into tasks (the flywheel).

1. Read traces from the generated plugin's trace-capture hook output: ${CLAUDE_PLUGIN_DATA}/traces/*.jsonl (tool args, results, errors per session). If no traces exist, say so and stop — mine has no other input; suggest confirming the trace-capture hook is installed.
2. Cluster failures (repeated errors, user corrections, wrong outputs) and propose each cluster as a candidate task via AskUserQuestion — same interview discipline: propose, recommend, WAIT.
3. For each accepted failure, author a task per references/task-authoring.md with the real failing input as instruction.md seed, a fixture reconstructing the state, and a reference solution showing the correct behavior. New tasks go to evals/tasks/ in the regression suite (they protect against a now-known failure).
4. A frozen suite that gains tasks is a NEW suite: finish by instructing the user to run /plugin-forge:arm-evals to re-gate and re-freeze (registry version bumps, scores stay comparable). If guard hooks block the writes, that is the freeze working — route the change through .forge/eval-change-requests.md as the guard message says.

Standing rules (all modes)

• Deterministic-first, always. A judge where a file check would do is a bug.
• Every task ships a reference solution. No exceptions — 0% pass usually means a broken task, and only the reference proves otherwise.
• Positive AND negative trigger cases for every auto-invoked component; one-sided suites train over-triggering.
• Golden answers, key-fact lists, and reference solutions live under reference/, which the harness stages OUTSIDE the agent-visible worktree. Never copy any of it into fixtures/ or instruction.md.
• Holdout tasks (evals-holdout/) are never named, quoted, or summarized in any session the builder loop can read.
• Track cost honestly: tracked_metrics are recorded, never graded; suite cost estimates go to the user before, not after, an expensive run.
• Never emit deprecated shapes: no legacy decision/approve hook JSON, no legacy mount-path skill references, and MCP tool names always in the full plugin-scoped form mcp__plugin_<plugin>_<server>__<tool> — never the short server-only form.

References

• references/task-authoring.md — task budget, schema, instruction grammar, fixtures, holdout, elicitation fixtures
• references/grader-menu.md — grader hierarchy, archetype routing, trigger + crowded target, simulate_user
• references/judge-authoring.md — modelgraded spec shape, judge rules, calibration handoff
• references/antipatterns.md — reward-hacking catalog with the mechanical lints L01–L16