Arm Evals

!cat .forge/state.json 2>/dev/null || echo NO_STATE

arm-evals — four gates, then the freeze

If the first line of this skill rendered as literal backtick syntax instead of JSON or NO_STATE (org policy disableSkillShellExecution blocks dynamic injection), read .forge/state.json with the Read tool before doing anything else.

A drafted suite is not yet a contract. It becomes one only after proving three things — its graders can recognize a correct solution, bare Claude cannot pass it, and its judges agree with a human — and then being made tamper-evident by a cryptographic freeze. Run the four gates strictly in order; each later gate is more expensive than the one before it, and an earlier failure invalidates later results.

State guard

Route on the injected state above. Refuse out-of-order execution and give the exact remediation command — never "help anyway".

Read plugin_dir from .forge/state.json — every gate command below needs it.

Before gate 1

• If a previous run crashed or state looks inconsistent, run forge-eval doctor first.
• Check .forge/eval-change-requests.md: review every pending entry with the user, apply or reject each, and record the decision in the file. Refuse to freeze over unreviewed requests.
• Cost honesty before starting: gate 2 executes the full suite headlessly (tasks x trials). Estimate from the task count, tell the user, and pass --max-cost-usd on every run. Use --dry-run once to validate wiring before spending anything, and --jobs N (max 8) to parallelize.

Gate 1 — REFERENCE GREEN

Every task's reference/ solution must pass that task's own graders, headlessly. For each suite in evals/registry.yaml:

forge-eval reference --suite <suite> --plugin-dir <plugin_dir>

Doctrine: a task whose own reference solution scores 0% is a broken task, not a hard task. The grader cannot recognize a known-correct answer, so it can never recognize the builder's. On any failure, fix the task or the grader (with the user for judgment calls), then re-run this gate for the affected tasks. Do not proceed to gate 2 with a red reference.

Run this gate first because it is the cheapest and because a broken grader corrupts every later gate's evidence.

Gate 2 — RED BASELINE

Run the full suite against the no-plugin target (targets/no-plugin.yaml, i.e. bare Claude with the plugin absent):

forge-eval red --suite <suite> --plugin-dir <plugin_dir> --max-cost-usd <X>

• Every task SHOULD fail here. A task that PASSES without the plugin is flagged "no signal" — it measures nothing the plugin adds. For each no-signal task, choose with the user:
  • Harden it: strengthen the end state, add a second verification signal, tighten a tolerance, or require an artifact only the plugin's workflow produces — while keeping the two-experts-same-verdict bar. Every hardened task must re-pass gate 1.
  • Drop it: delete the task if hardening would make it artificial.
• Store the baseline: the red run's runs/<ts>/summary.json is the with/without-plugin denominator that verify later reports the improvement delta against. Never delete that run directory; cite its path in the arming report.
• Trigger-suite tasks trivially show no activation under the no-plugin target; that is expected and is not a no-signal finding — trigger precision/recall is measured later by verify under bare and crowded targets.

Gate 3 — JUDGE CALIBRATION

For every judge spec under evals/judges/ referenced by any task grader:

1. Labels first. Calibration needs 10–20 human-labeled transcripts per judge dimension, drawn from the gate 1 and gate 2 runs (runs/<ts>/transcripts/). If evals/labels/labels.jsonl is missing or too thin, sample the transcripts, present them to the user for hand-labeling in the scaffolded labels.jsonl format, and WAIT. Never fill labels.jsonl by model judgment — a model-labeled golden set makes the agreement metric circular and voids this gate entirely.

2. Measure agreement:

   forge-eval metaeval --plugin-dir <plugin_dir>
   

   This reports per-judge agreement between the judge's verdicts and the human labels.

3. Agreement >= 0.85 → the judge passes. Below 0.85, run the Align-Evals iteration loop:

   • sort disagreements worst-first and read those transcripts alongside the judge's own rationale/evidence output;
   • add explicit negative criteria to the judge prompt for each observed false-pass pattern;
   • insert the corrected cases into the judge prompt as few-shot examples;
   • re-version the judge spec (bump its version marker so scores stay attributable to a judge version), re-run metaeval, and re-baseline agreement per judge version;
   • repeat until agreement reaches 0.85 or the judge is replaced.

4. Judges run on a pinned cheap model that is not the target model. Changing the pinned judge model invalidates calibration — re-run metaeval after any model change.

Refusal is mandatory here. If any judge is uncalibrated (no labels, too few labels, or agreement below 0.85), refuse to freeze and say exactly why: an uncalibrated judge silently converts "measured" into "vibes" — its scores cannot bind a build loop, and a green result would be unfalsifiable. Do not accept deleting the judge grader as a workaround: the PDR capability row that judge verifies becomes unobservable, and unverifiable means unbuildable. The legitimate paths are labeling the transcripts (preferred) or replacing the judge with a deterministic two-signal check — which is a contract change requiring a change-request entry and a spec-critic re-review.

Gate 4 — FREEZE

Only when gates 1–3 are green against the exact current tree. If any file under evals/ changed while running the gates (hardened tasks, iterated judges, added labels), re-run gate 1 for edited tasks and gate 3 for edited judges before freezing.

1. Determine the suite version N. No .forge/freeze.json → v1. Otherwise recompute the tree hash (step 3's pipeline): identical to the recorded sha256 means nothing changed (report "already armed", stop); different means N = previous N + 1. Any suite edit auto-bumps N — that is what keeps scores comparable across suite revisions.

2. Bump the registry BEFORE hashing. Update every alias in evals/registry.yaml to <suite>.<split>.v<N>. The registry lives inside evals/, so editing it after computing the hash would invalidate the freeze immediately.

3. Compute the tree sha256 from the project root:

   find evals -type f -print0 | LC_ALL=C sort -z | xargs -0 shasum -a 256 | shasum -a 256 | awk '{print $1}'
   

   This pipeline is the canonical TREE HASH CONTRACT documented in ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/sweep-evals.sh and mirrored by tree_sha256 in the harness (scripts/harness/run.py). Never substitute a different hashing recipe — the Stop-hook sweep compares against exactly this value during the build, and a divergent recipe would either block every stop or detect nothing.

4. Write .forge/freeze.json (it lives outside evals/, so it does not perturb the hash):

   {"sha256": "<tree-hash>", "suite_version": "v<N>", "frozen_at": "<ISO8601 UTC>"}
   

5. Update .forge/state.json: set phase to "armed" and updated to the current ISO8601 timestamp; preserve every other field.

6. Holdout check: confirm evals-holdout/ exists OUTSIDE evals/ (project root). It is deliberately excluded from the freeze hash and is never seeded into the build worktree — it is the anti-overfit tranche the builder must never see. If build-evals created no holdout, flag it to the user before proceeding.

Re-arming after any suite edit

Triggers: a triage report classifying a failure as grader-error; an approved entry in .forge/eval-change-requests.md; any manual edit under evals/ (labels included — they are part of the hashed tree); a judge-model change.

Procedure: review pending change requests with the user, apply the edits, re-run the affected gates (gate 1 for every edited task, gate 2 when tasks were hardened/added/dropped, gate 3 for any touched judge), then gate 4 — the version bump happens automatically because the tree hash differs. Re-arming from smoke, verify, or ship regresses phase to armed: prior green results were measured against the old suite version and no longer bind.

Refusal conditions (all hard)

Refuse to freeze while any of these holds, stating which gate failed, why the freeze would be unsound, and the exact next action:

1. Any reference solution failing its own graders (gate 1).
2. Any unresolved no-signal task (gate 2).
3. Any judge with agreement below 0.85, or with missing/insufficient human labels (gate 3).
4. labels.jsonl populated by the model instead of the human — circular calibration (gate 3).
5. Unreviewed entries in .forge/eval-change-requests.md.
6. phase: building — never re-freeze under a live build.

Handoff

Report the arming result on one line plus the next command, for example:

ARMED: suite_version=v2 sha256=3fa9c1d2e0b4... tasks=24 judges=2 (agreement 0.91, 0.88) red_baseline=runs/2026-08-02T14-11-05/
Next: /plugin-forge:build-loop

The suite is now the contract: during the build it is mechanically read-only (PreToolUse guard + Stop-hook sweep, holding even under --dangerously-skip-permissions), and the only legitimate way to change it is a change request plus a human re-running this skill.