Verify

Verify — smoke → pass^k → holdout → triggers → budget → triage

Current forge state: !cat .forge/state.json 2>/dev/null || echo NO_STATE

If the line above shows a policy notice instead of JSON (org setting disableSkillShellExecution), read .forge/state.json with the Read tool before continuing.

Phase gate (hard)

Proceed only when phase is smoke (start at Step 1) or verify (smoke already passed — resume at Step 2). For any other state, run NOTHING and reply with exactly the remediation below, then stop:

Read plugin_name and plugin_dir from the injected state and use them in every command below. If the state file looks corrupted or stale, run forge-eval doctor and re-check before deciding.

Ground rules

• The suite is the contract. Never edit anything under evals/ or evals-holdout/ during verification. Propose needed suite changes in .forge/eval-change-requests.md; only /plugin-forge:arm-evals executes them (version bump + re-freeze).
• Plugin edits are allowed — that is what verification feedback is for — but they invalidate earlier green steps: after a description edit, re-run Step 4 (and re-check Step 5); after any behavior edit, re-run Steps 2–6.
• Run suites in background Bash. A real suite exceeds foreground Bash timeouts. Launch every forge-eval run with run_in_background: true; when the job completes, read its exit code and the final FORGE_EVAL: stdout line (exit 0 iff RESULT=PASS). Progress is resumable from the run's progress file under runs/<ts>/.
• Cost cap every run. Pass --max-cost-usd <X> on every invocation, with X from the CONTRACT.md cost sheet (ask the user if the sheet has no verify budget). Report cumulative cost_usd per step.
• Infra is not failure. Trials recorded timeout (exit 143) or unverified (api_retry) are excluded from pass rates. Re-run affected tasks with --task <ID>; never let infra noise flip a gate in either direction.
• Fresh evidence only. Never reuse build-loop run artifacts as verification evidence; every number reported below comes from runs started by this skill.

Step 1 — Smoke matrix (entered at phase smoke)

Catch cheap failures before paying for trials:

forge-eval smoke --plugin-dir <plugin_dir>

The smoke matrix asserts, in order:

1. System/init gate — a claude -p --bare --plugin-dir <plugin_dir> session starts with plugin_errors and mcp_server_errors empty or omitted, and <plugin_name> present in plugins[].
2. Per-skill probes — every user-invocable skill invoked once via claude -p --bare "/<plugin_name>:<skill> <fixture args>", asserting a non-error result.
3. Hook fixture pipes — each recorded event fixture piped into each hook handler, asserting exit code AND output-channel discipline (stderr + exit 2 to block; never exit-2 mixed with JSON output).
4. Trigger sanity — 3 should-trigger phrasings per model-invocable skill in fresh sessions, counting activations.

Gate: report status ALL_PASS. Infra-flagged probes are re-run, not counted.

• On a real failure: a plugin defect escaped the build loop. Set phase back to armed (snippet in State transitions; regression rule R4 in phase-gates.md), run /plugin-forge:build-loop, then restart verification here.
• On ALL_PASS: set phase to verify and continue.

Step 2 — Full suite, fresh, at three trials

Run capability and regression against the default target:

forge-eval run --suite capability --plugin-dir <plugin_dir> --target default --trials 3 --max-cost-usd <X>
forge-eval run --suite regression --plugin-dir <plugin_dir> --target default --trials 3 --max-cost-usd <X>

Gates:

• Both FORGE_EVAL: lines end RESULT=PASS (exit code 0).
• The headline number is capability pass^3 from the scoreboard line — report it verbatim, alongside the suite version (version=vN must match .forge/freeze.json).
• Regression is expected ≈100%; any regression failure goes to Step 6 triage before proceeding.

Step 3 — Holdout tranche

The holdout tranche lives in evals-holdout/ and was never seeded into the build worktree — the builder has never seen these tasks or their graders. This is the anti-overfit check (point the harness at the holdout tree with --evals-dir, using the suite name from evals-holdout/suites/):

forge-eval run --suite <suite> --plugin-dir <plugin_dir> --evals-dir evals-holdout --target default --trials 3 --max-cost-usd <X>

Gate: RESULT=PASS. If the holdout score sits materially below Step 2 (e.g. a pass^3 gap greater than 0.2), treat it as overfitting to visible graders: dispatch triage (Step 6) immediately, expect task hardening via /plugin-forge:arm-evals, and re-enter the build loop. The honest claim after this step is "green + holdout + triaged" — never "correct".

Step 4 — Trigger suite on BOTH targets

forge-eval run --suite triggers --plugin-dir <plugin_dir> --target default --max-cost-usd <X>
forge-eval run --suite triggers --plugin-dir <plugin_dir> --target crowded --max-cost-usd <X>

• default = clean --bare session: the optimistic ceiling.
• crowded = distractor plugins loaded via repeated --plugin-dir: realistic 1%-listing-budget pressure. This is the honest number.

Report precision and recall for BOTH targets — never only default. Gate: crowded-target numbers meet the CONTRACT.md trigger targets; if the contract sets none, require precision 1.0 and recall ≥ 0.8 and surface anything lower to the user. A crowded-only recall miss is a description problem: tweak the component's description keywords (a plugin edit — allowed), then re-run both targets and re-check Step 5, because description edits change always-on cost.

Step 5 — Token budget (always-on cost)

claude plugin details <plugin_name>

(If the name does not resolve because the plugin is not installed, run the command from a session started with claude --plugin-dir <plugin_dir>.)

Compare the always-on token cost — descriptions charged to every session — against the CONTRACT.md cost-sheet budget. Over budget: trim descriptions or demote components (user-invocable: false for knowledge, disable-model-invocation: true for user-gated actions) per the contract, then re-run Step 4 since descriptions changed. Record both always-on and on-invoke figures in the report.

Step 6 — Triage dispatch

Dispatch the plugin-forge:triage agent over the newest runs/<ts>/ directories from Steps 2–4. Have it sample every failing trial plus 1–2 passing trials per failed task (dual-trajectory inspection). It writes runs/<ts>/triage.md, labeling each sampled failure {agent-error | grader-error | task-ambiguity} with a recommended fix location. Route by classification:

Ship-ready

Declare verification green only when ALL of:

1. Smoke ALL_PASS.
2. Capability + regression fresh at --trials 3: RESULT=PASS, pass^3 headline recorded.
3. Holdout RESULT=PASS with no overfit gap.
4. Trigger precision/recall reported for default AND crowded; crowded meets targets.
5. Always-on token cost within the contract budget.
6. Triage report exists with zero unresolved grader-error / task-ambiguity findings.

Then:

1. Write runs/<latest-ts>/verify-report.md: the six gates with their numbers, run timestamps, suite version from .forge/freeze.json, and total cost.
2. Set phase to ship (snippet below).
3. Tell the user verification is green and the next step is /plugin-forge:ship. Ship is user-gated (disable-model-invocation: true) — never attempt to invoke it; the user runs it.

State transitions

This skill writes .forge/state.json exactly twice: verify after smoke ALL_PASS, ship after all six gates. Regressions to armed (defect routing above) use the same snippet.

python3 - <new-phase> <<'EOF'
import json, sys, datetime, pathlib
p = pathlib.Path('.forge/state.json')
s = json.loads(p.read_text())
s['phase'] = sys.argv[1]
s['updated'] = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')
p.write_text(json.dumps(s, indent=2) + '\n')
print('forge phase ->', s['phase'])
EOF