# Verify

> Verify a forge-built plugin before shipping: smoke matrix, fresh pass^k trials, holdout tranche, trigger evals on default AND crowded targets, token budget check, and triage of sampled failures. Use after a forge build loop has gone green (.forge/state.json phase smoke or verify) and before /plugin-forge:ship.

- Skill: `aznatkoiny/verify` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aznatkoiny/verify`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aznatkoiny/verify/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Finance & Business
- Author: aznatkoiny (https://skillmd.com/u/aznatkoiny)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aznatkoiny/verify

---


# Verify — smoke → pass^k → holdout → triggers → budget → triage

Current forge state: !`cat .forge/state.json 2>/dev/null || echo NO_STATE`

If the line above shows a policy notice instead of JSON (org setting
`disableSkillShellExecution`), read `.forge/state.json` with the Read tool
before continuing.

## Phase gate (hard)

Proceed only when `phase` is `smoke` (start at Step 1) or `verify` (smoke
already passed — resume at Step 2). For any other state, run NOTHING and reply
with exactly the remediation below, then stop:

| Injected state | Reply and stop |
|---|---|
| NO_STATE | "No forge run in this project. Run `/plugin-forge:forge` to start one." |
| `interview` | "Interview incomplete. Run `/plugin-forge:forge`." |
| `contract` | "No approved contract yet. Run `/plugin-forge:write-contract`." |
| `evals` | "Suite not armed. Run `/plugin-forge:build-evals`, then `/plugin-forge:arm-evals`." |
| `armed` | "Nothing built yet. Run `/plugin-forge:build-loop`." |
| `building` | "Build loop has not gone green. Run `/plugin-forge:build-loop` to resume it." |
| `ship` | "Verification is already green. Run `/plugin-forge:ship`." |

Read `plugin_name` and `plugin_dir` from the injected state and use them in
every command below. If the state file looks corrupted or stale, run
`forge-eval doctor` and re-check before deciding.

## Ground rules

- **The suite is the contract.** Never edit anything under `evals/` or
  `evals-holdout/` during verification. Propose needed suite changes in
  `.forge/eval-change-requests.md`; only `/plugin-forge:arm-evals` executes
  them (version bump + re-freeze).
- **Plugin edits are allowed** — that is what verification feedback is for —
  but they invalidate earlier green steps: after a description edit, re-run
  Step 4 (and re-check Step 5); after any behavior edit, re-run Steps 2–6.
- **Run suites in background Bash.** A real suite exceeds foreground Bash
  timeouts. Launch every `forge-eval run` with `run_in_background: true`; when
  the job completes, read its exit code and the final `FORGE_EVAL:` stdout
  line (exit 0 iff `RESULT=PASS`). Progress is resumable from the run's
  progress file under `runs/<ts>/`.
- **Cost cap every run.** Pass `--max-cost-usd <X>` on every invocation, with
  X from the CONTRACT.md cost sheet (ask the user if the sheet has no verify
  budget). Report cumulative `cost_usd` per step.
- **Infra is not failure.** Trials recorded `timeout` (exit 143) or
  `unverified` (api_retry) are excluded from pass rates. Re-run affected tasks
  with `--task <ID>`; never let infra noise flip a gate in either direction.
- **Fresh evidence only.** Never reuse build-loop run artifacts as
  verification evidence; every number reported below comes from runs started
  by this skill.

## Step 1 — Smoke matrix (entered at phase `smoke`)

Catch cheap failures before paying for trials:

```
forge-eval smoke --plugin-dir <plugin_dir>
```

The smoke matrix asserts, in order:

1. **System/init gate** — a `claude -p --bare --plugin-dir <plugin_dir>`
   session starts with `plugin_errors` and `mcp_server_errors` empty or
   omitted, and `<plugin_name>` present in `plugins[]`.
2. **Per-skill probes** — every user-invocable skill invoked once via
   `claude -p --bare "/<plugin_name>:<skill> <fixture args>"`, asserting a
   non-error result.
3. **Hook fixture pipes** — each recorded event fixture piped into each hook
   handler, asserting exit code AND output-channel discipline (stderr + exit 2
   to block; never exit-2 mixed with JSON output).
4. **Trigger sanity** — 3 should-trigger phrasings per model-invocable skill
   in fresh sessions, counting activations.

Gate: report status **ALL_PASS**. Infra-flagged probes are re-run, not counted.

- On a real failure: a plugin defect escaped the build loop. Set phase back to
  `armed` (snippet in State transitions; regression rule R4 in
  phase-gates.md), run `/plugin-forge:build-loop`, then restart verification
  here.
- On ALL_PASS: set phase to `verify` and continue.

## Step 2 — Full suite, fresh, at three trials

Run capability and regression against the default target:

```
forge-eval run --suite capability --plugin-dir <plugin_dir> --target default --trials 3 --max-cost-usd <X>
forge-eval run --suite regression --plugin-dir <plugin_dir> --target default --trials 3 --max-cost-usd <X>
```

Gates:

- Both `FORGE_EVAL:` lines end `RESULT=PASS` (exit code 0).
- The headline number is capability `pass^3` from the scoreboard line — report
  it verbatim, alongside the suite version (`version=vN` must match
  `.forge/freeze.json`).
- Regression is expected ≈100%; any regression failure goes to Step 6 triage
  before proceeding.

## Step 3 — Holdout tranche

The holdout tranche lives in `evals-holdout/` and was never seeded into the
build worktree — the builder has never seen these tasks or their graders. This
is the anti-overfit check (point the harness at the holdout tree with
`--evals-dir`, using the suite name from `evals-holdout/suites/`):

```
forge-eval run --suite <suite> --plugin-dir <plugin_dir> --evals-dir evals-holdout --target default --trials 3 --max-cost-usd <X>
```

Gate: `RESULT=PASS`. If the holdout score sits materially below Step 2 (e.g. a
pass^3 gap greater than 0.2), treat it as overfitting to visible graders:
dispatch triage (Step 6) immediately, expect task hardening via
`/plugin-forge:arm-evals`, and re-enter the build loop. The honest claim after
this step is "green + holdout + triaged" — never "correct".

## Step 4 — Trigger suite on BOTH targets

```
forge-eval run --suite triggers --plugin-dir <plugin_dir> --target default --max-cost-usd <X>
forge-eval run --suite triggers --plugin-dir <plugin_dir> --target crowded --max-cost-usd <X>
```

- `default` = clean `--bare` session: the optimistic ceiling.
- `crowded` = distractor plugins loaded via repeated `--plugin-dir`: realistic
  1%-listing-budget pressure. This is the honest number.

Report precision and recall for BOTH targets — never only default. Gate:
crowded-target numbers meet the CONTRACT.md trigger targets; if the contract
sets none, require precision 1.0 and recall ≥ 0.8 and surface anything lower
to the user. A crowded-only recall miss is a description problem: tweak the
component's description keywords (a plugin edit — allowed), then re-run both
targets and re-check Step 5, because description edits change always-on cost.

## Step 5 — Token budget (always-on cost)

```
claude plugin details <plugin_name>
```

(If the name does not resolve because the plugin is not installed, run the
command from a session started with `claude --plugin-dir <plugin_dir>`.)

Compare the always-on token cost — descriptions charged to every session —
against the CONTRACT.md cost-sheet budget. Over budget: trim descriptions or
demote components (`user-invocable: false` for knowledge,
`disable-model-invocation: true` for user-gated actions) per the contract,
then re-run Step 4 since descriptions changed. Record both always-on and
on-invoke figures in the report.

## Step 6 — Triage dispatch

Dispatch the `plugin-forge:triage` agent over the newest `runs/<ts>/`
directories from Steps 2–4. Have it sample every failing trial plus 1–2
passing trials per failed task (dual-trajectory inspection). It writes
`runs/<ts>/triage.md`, labeling each sampled failure
`{agent-error | grader-error | task-ambiguity}` with a recommended fix
location. Route by classification:

| Finding | Action |
|---|---|
| `grader-error` | STOP verification. Run `/plugin-forge:arm-evals` (fix the grader, bump the suite to vN+1, re-freeze), then repeat from Step 2 — scores are not comparable across suite versions. |
| `task-ambiguity` | Repair the task via `/plugin-forge:build-evals`, re-freeze via `/plugin-forge:arm-evals`, then repeat from Step 2. |
| `agent-error` and any gate is RESULT=FAIL | Set phase to `armed` (regression rule R4 in phase-gates.md), run `/plugin-forge:build-loop`, then re-verify from Step 1. |
| `agent-error` only, all gates PASS | Tolerated residue within pass^k. Record it in the report and proceed. |

## Ship-ready

Declare verification green only when ALL of:

1. Smoke ALL_PASS.
2. Capability + regression fresh at `--trials 3`: RESULT=PASS, pass^3 headline
   recorded.
3. Holdout RESULT=PASS with no overfit gap.
4. Trigger precision/recall reported for default AND crowded; crowded meets
   targets.
5. Always-on token cost within the contract budget.
6. Triage report exists with zero unresolved `grader-error` /
   `task-ambiguity` findings.

Then:

1. Write `runs/<latest-ts>/verify-report.md`: the six gates with their
   numbers, run timestamps, suite version from `.forge/freeze.json`, and total
   cost.
2. Set phase to `ship` (snippet below).
3. Tell the user verification is green and the next step is
   `/plugin-forge:ship`. Ship is user-gated (`disable-model-invocation:
   true`) — never attempt to invoke it; the user runs it.

## State transitions

This skill writes `.forge/state.json` exactly twice: `verify` after smoke
ALL_PASS, `ship` after all six gates. Regressions to `armed` (defect routing
above) use the same snippet.

```
python3 - <new-phase> <<'EOF'
import json, sys, datetime, pathlib
p = pathlib.Path('.forge/state.json')
s = json.loads(p.read_text())
s['phase'] = sys.argv[1]
s['updated'] = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')
p.write_text(json.dumps(s, indent=2) + '\n')
print('forge phase ->', s['phase'])
EOF
```