# Build Evals

> Generate a runnable eval suite (tasks, graders, judges, targets) for a Claude Code plugin BEFORE it is built. Use for "build evals", "eval suite for my plugin", "test my plugin", "retrofit evals", "mine traces into tasks". In the forge pipeline this is the evals phase; retrofit and mine modes run standalone against any existing plugin.

- Skill: `aznatkoiny/build-evals` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add aznatkoiny/build-evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aznatkoiny/build-evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: aznatkoiny (https://skillmd.com/u/aznatkoiny)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aznatkoiny/build-evals

---


# build-evals — the eval builder

!`cat .forge/state.json 2>/dev/null || echo NO_STATE`

Generate the complete `evals/` tree for a target plugin. In `new` mode the plugin does not
exist yet: tasks derive from `design/CONTRACT.md` (the PDR), not from code. The suite this
skill emits becomes the definition of "done" for the entire build — treat every task as a
contract clause, not a test.

This skill drafts and lints the suite. It never arms, freezes, or scores it — those are
`/plugin-forge:arm-evals` and `bin/forge-eval`. It is model-invocable by design (decision D2):
the forge conductor chains it via the Skill tool; safety comes from the state guard below,
not from invocation control.

## State guard (new mode only)

Apply this table to the injected state line above before doing anything else in `new` mode
(also the default when no argument is given). `retrofit` and `mine` skip this guard entirely —
they run standalone against any plugin, with or without a forge pipeline (see their sections).

| Injected state | Action |
|---|---|
| `NO_STATE` | REFUSE. Say: "No forge pipeline in this project. Run `/plugin-forge:forge` to start one, or use `/plugin-forge:build-evals retrofit <plugin-path>` for an existing plugin." Stop. |
| `phase` is `interview` or `contract` | REFUSE. Say: "The contract is not approved yet — evals derive from it. Run `/plugin-forge:forge resume`." Stop. |
| `phase` is `evals` | Proceed with new mode. |
| `phase` is `armed`, `building`, `smoke`, `verify`, or `ship` | REFUSE edits. The suite is frozen (sha256 in `.forge/freeze.json`); guard hooks will block writes anyway. Say: "The suite is frozen. Append the desired change to `.forge/eval-change-requests.md`, then run `/plugin-forge:arm-evals` to apply it, re-gate, and re-freeze as a new version." Stop. |

## Mode: new (pipeline phase)

Work through the steps in order. Load the referenced file before executing its step —
each reference is binding, not advisory.

### 1. Map the surface from the contract

Read `design/CONTRACT.md`. Extract, per capability row:

- the chosen composition (skills, hooks, agents, MCP servers, scripts) and its archetype
- the **C5 Testability** column: grader type, observable signal, and planned task IDs —
  this column IS the task backlog; build-evals exists to make those IDs real
- the **C1 Trigger** column: every auto-invoked component (model-invoked skill, hook,
  agent with proactive description) needs positive AND negative trigger tasks
- external dependencies (MCP servers, APIs, corpora) and whether the contract chose live
  or mock for each

Also read `design/INTENT.md` for the runtime-environment answers (headless/cloud/local)
— they constrain what graders can observe.

### 2. Interview — propose clusters, recommend, WAIT

Never one-shot the suite (interviewing beats one-shot generation — LangChain's central
empirical finding). Use AskUserQuestion to:

1. Propose 3–5 testable ability clusters derived from the PDR rows, each with a one-line
   description, its archetype, and a rough task count. Recommend one ordering and say why.
   WAIT for the user's choice before generating anything.
2. For every external dependency, ask live-vs-mock. Default recommendation: mock fixture
   stubs for write-heavy, paid, or rate-limited tools; live only for cheap read-only calls
   the user explicitly accepts. Record the decision per dependency — the harness stores
   which mode each result came from.

For scientific or business domains where fixture corpora need domain-real content, offer
to run the `/plugin-forge:skill-forge` research pipeline to ground fixtures in real sources instead of
inventing plausible-looking data (decision D10).

### 3. Scaffold

Copy the scaffold from `${CLAUDE_PLUGIN_ROOT}/templates/evals/` into the target plugin as
`evals/`, and create the sibling `evals-holdout/` next to it (NOT inside `evals/`). The
scaffold provides `registry.yaml`, `targets/{default,no-plugin,crowded}.yaml`,
`suites/{capability,regression,triggers}.yaml`, `judges/{fact,rubric,battle}.yaml`,
`labels/labels.jsonl`, and `tasks/_example/`. Delete `tasks/_example/` after real tasks
exist.

Then vendor the harness runner (stdlib-only, so the suite outlives plugin-forge):

```
mkdir -p <plugin_dir>/evals/bin && cp "${CLAUDE_PLUGIN_ROOT}/scripts/harness/run.py" <plugin_dir>/evals/bin/run.py && chmod +x <plugin_dir>/evals/bin/run.py
```

### 4. Author tasks

Follow `references/task-authoring.md` for everything about tasks: the exact `task.yaml`
schema, the 20–50 task budget, the two-experts bar, the REQUIRED reference solution, the
goal-condition grammar for `instruction.md`, fixture rules, the holdout split, and
`elicitation_answers`. Every task ID planned in the contract's C5 column must exist in
`evals/tasks/` or `evals-holdout/tasks/` when this step ends.

### 5. Choose graders

Follow `references/grader-menu.md`: deterministic-first hierarchy (workspace → transcript
→ state_check → judge → trigger, plus the simulate_user driver), archetype routing, the
mandatory numeric `tolerance`, the never-assert-tool-ORDER rule, and the crowded-listing
requirement for trigger tasks.

### 6. Author judges

Only where the grader menu genuinely runs out of deterministic options. Follow
`references/judge-authoring.md`: modelgraded-spec YAML shape, one rubric dimension per
isolated judge call, Unknown always legal, pinned cheap judge model different from the
target model, anti-echo phrasing. Seed `evals/labels/labels.jsonl` with the unlabeled
sample skeleton — calibration itself is arm-evals' gate, not this skill's.

### 7. Generate the elicitation auto-answer hook (when applicable)

If the target plugin bundles or drives an MCP server that uses elicitation, headless eval
runs will hang or die — no dialog exists under `claude -p`. For every such server:

- set `elicitation_answers` in each task.yaml that exercises it (see task-authoring.md)
- emit `evals/hooks/elicitation-auto-answer.sh` (bash, `set -euo pipefail`, executable) and
  `evals/hooks/elicitation-settings.json` registering it as an exec-form Elicitation hook:
  `"command": "${CLAUDE_PLUGIN_ROOT}/evals/hooks/elicitation-auto-answer.sh"` — harness runs
  merge this settings file; production users never load it. The script reads the elicitation
  request JSON from stdin, looks up the requested fields in
  `.forge-eval/elicitation-answers.json` (staged into the workspace by the harness from the
  task's `elicitation_answers`), and prints the matching answers object to stdout with exit 0.
  No match → one-line stderr message and exit 2, so the trial fails loudly instead of hanging.

### 8. Lint the suite

Run every mechanical lint in `references/antipatterns.md` (L01–L16) and fix all findings
before proceeding. L01 (numeric tolerance) and L02 (task-text/grader consistency) are the
builder meta-eval contract: plugin-forge's own dogfood suite feeds this skill adversarial
specs engineered to reproduce the CORE-Bench and METR grader bugs, and passes only if the
emitted suite provably cannot contain them. Do not hand a suite to arm-evals with any lint
red.

### 9. Smoke-run drafts and hand off

An eval that has never run is not delivered:

- Run `forge-eval reference --task <ID>` for at least 3 representative tasks (one per
  archetype used) and show the user the graded output. Use `forge-eval run --dry-run` for a
  cost estimate of the full suite; state it plainly — trials × tasks × judges is real money,
  and `--max-cost-usd` exists for a reason.
- Present the suite summary: task count per suite (capability/regression/triggers), holdout
  count, judge dimensions, live-vs-mock map, estimated full-run cost.
- Tell the user the exact next command: `/plugin-forge:arm-evals` — the four arming gates
  (reference-green, red baseline, judge calibration, freeze) all run there.

Do not modify `.forge/state.json`. The phase advances to `armed` only when arm-evals
freezes the suite.

## Mode: retrofit <plugin-path>

Standalone — no forge pipeline, no state guard, works on any installed or in-repo plugin.

1. Map the plugin surface from files (there is no PDR): every `skills/*/SKILL.md` and
   command frontmatter (descriptions = trigger contracts), `agents/*.md`, `hooks/hooks.json`
   (matchers = firing contracts), `.claude-plugin/plugin.json`, `.mcp.json`. Derive one
   ability cluster per user-facing capability and one trigger axis per auto-invoked
   component. MCP tool checks must use plugin-scoped names:
   `mcp__plugin_<plugin>_<server>__<tool>`.
2. Then run steps 2–9 above unchanged, writing `evals/` and `evals-holdout/` into the
   plugin's own directory. Where step 1 of new mode says CONTRACT.md, use the surface map;
   flag any capability whose observable signal is unclear as "unverifiable — needs a design
   decision" instead of inventing a grader for it.
3. Recommend arming: retrofit suites get the same four gates via
   `/plugin-forge:arm-evals <plugin-path>` before their scores mean anything.

## Mode: mine

Standalone — converts captured production failures into tasks (the flywheel).

1. Read traces from the generated plugin's trace-capture hook output:
   `${CLAUDE_PLUGIN_DATA}/traces/*.jsonl` (tool args, results, errors per session). If no
   traces exist, say so and stop — mine has no other input; suggest confirming the
   trace-capture hook is installed.
2. Cluster failures (repeated errors, user corrections, wrong outputs) and propose each
   cluster as a candidate task via AskUserQuestion — same interview discipline: propose,
   recommend, WAIT.
3. For each accepted failure, author a task per `references/task-authoring.md` with the
   real failing input as `instruction.md` seed, a fixture reconstructing the state, and a
   reference solution showing the correct behavior. New tasks go to `evals/tasks/` in the
   regression suite (they protect against a now-known failure).
4. A frozen suite that gains tasks is a NEW suite: finish by instructing the user to run
   `/plugin-forge:arm-evals` to re-gate and re-freeze (registry version bumps, scores stay
   comparable). If guard hooks block the writes, that is the freeze working — route the
   change through `.forge/eval-change-requests.md` as the guard message says.

## Standing rules (all modes)

- Deterministic-first, always. A judge where a file check would do is a bug.
- Every task ships a reference solution. No exceptions — 0% pass usually means a broken
  task, and only the reference proves otherwise.
- Positive AND negative trigger cases for every auto-invoked component; one-sided suites
  train over-triggering.
- Golden answers, key-fact lists, and reference solutions live under `reference/`, which
  the harness stages OUTSIDE the agent-visible worktree. Never copy any of it into
  `fixtures/` or `instruction.md`.
- Holdout tasks (`evals-holdout/`) are never named, quoted, or summarized in any session
  the builder loop can read.
- Track cost honestly: `tracked_metrics` are recorded, never graded; suite cost estimates
  go to the user before, not after, an expensive run.
- Never emit deprecated shapes: no legacy decision/approve hook JSON, no legacy mount-path
  skill references, and MCP tool names always in the full plugin-scoped form
  `mcp__plugin_<plugin>_<server>__<tool>` — never the short server-only form.

## References

- `references/task-authoring.md` — task budget, schema, instruction grammar, fixtures, holdout, elicitation fixtures
- `references/grader-menu.md` — grader hierarchy, archetype routing, trigger + crowded target, simulate_user
- `references/judge-authoring.md` — modelgraded spec shape, judge rules, calibration handoff
- `references/antipatterns.md` — reward-hacking catalog with the mechanical lints L01–L16