Run a plugin eval suite
claude plugin eval runs each case with the plugin and without it, grades each run, and writes
aggregate-result.json. This skill does what the CLI does not, all before money is spent: checks
that this machine can run the suite, prices the run, routes a target that is not a plugin, and
reads the delta. It never re-implements grading and never simulates a run or a score.
Actions
| Action |
What happens |
preflight <target> |
Print the report below and stop. No CLI invocation, no spend |
validate [<eval-dir>] |
Run the static case validator only. No model call |
run <target> (default) |
Preflight, validate, print the estimate, then invoke the CLI |
read <json> |
Read a written aggregate-result.json in the order that keeps a delta honest, then remove the sandboxes its run kept (sandboxes.md) |
ci |
Answer from the CI section below |
init [<name>] |
Scaffold a suite: claude plugin eval init --bare <name> where there is no terminal |
This page holds every rule the actions use; answer from it. Each upstream specific it states has a
row under Sources. Background for a human reader:
case-authoring.md, reading-results.md
(JSON fields), ci.md (workflow file, exit codes, parser), and
records.md (the observations behind each source row). Tuning one case
after a read, or confirming a change, is not an action and is answered from
iterating.md, never from this page alone.
Preflight
Print every field, in order, before anything else (leftover_sandboxes and tmp_free: probe; eval_tmpdir: Sandbox start):
cli_version: <claude --version>
floor_met: <true | false>
git_version: <git --version | none>
platform: <windows | wsl2 | linux | darwin | other>
sandbox_backend: <present | absent | not-ready | unknown> (missing: <binaries that did not resolve> | links: <paths>)
target_type: <plugin | wrapped-skill | wrapped-agent | rules>
suite_tools: <read-only | the gated tools the cases request>
same_model: <yes | no | unknown | off> (tested <model>, judge <model>)
estimate_usd: roughly <n> (cases x runs x arms, plus judge calls)
ceiling: <n> USD | unlimited
leftover_sandboxes: <n> sandboxes (<total size> in <tmp>) | unknown
tmp_free: <free space on the temp dir's filesystem> | unknown
eval_tmpdir: <dir> (<n> characters) | too long: <dir> (<n> characters) | n/a
eval_tmpdir is the <dir> the run step will pass as TMPDIR on Linux and WSL2, else n/a.
Refuse before any spend when it is longer than 24 characters, the longest length measured to
work (Sandbox start), and any case requests Bash or PowerShell; the route is a shorter <dir>, as in Run step 3.
Preflight removes nothing and blocks on neither leftover_sandboxes nor tmp_free. It prints no removal command either: <tmp> can be shared with another session's run, whose live or kept sandboxes a glob over <tmp>/claude-eval-* would delete. When leftover_sandboxes is above zero, point at the removal by results file instead (sandboxes.md). Report an unreadable path as unknown, never 0.
Version floor: Claude Code 2.1.269. Never assert that the command is installed: read claude --version and let the number decide. floor_met: false stops the run and reports the
floor; nothing else is worth doing on a binary that cannot execute a case. plugin eval is currently in early access means an older binary: updating fixes it, no sign-up. plugin eval is currently unavailable means Anthropic has switched the command off for now, possibly by account
or context: nothing local fixes it and there is no access to request, so wait and retry.
Git floor: fetch https://code.claude.com/docs/en/plugin-evals#requirements live for it. A
git below it sets floor_met: false and names the version on the git_version line; no git
reports git_version: none and does not block.
Sandbox backend
No CLI string reports the backend, so detection is platform-shaped. When the user states their
platform, answer for that platform and take platform from what they said. Never infer it from
this session's own host, which may not be the machine that will run the eval.
| Observation |
sandbox_backend |
Native Windows (no WSL), or WSL1 (wsl.exe -l -v lists the distribution at VERSION 1) |
absent |
Linux, or WSL2 (wsl.exe -l -v lists the distribution at VERSION 2; /proc/version naming microsoft only says the host is WSL, since a custom WSL2 kernel can carry any release string), and every package the sandboxing page lists resolves (command -v bwrap, command -v socat) |
present |
| Linux or WSL2, and any of them is missing |
absent, naming each missing binary |
| macOS |
present |
| Anything else |
unknown, treated as absent for the refusal below |
Then, when the table gives present, a case requests Bash or PowerShell, and
find "${DOCKER_CONFIG:-$HOME/.docker}/" -mindepth 1 -type l (read-only) prints a path, override
to not-ready, naming each path and the CLI's reason: the sandbox cannot reliably exclude a store
with a link inside it (the store's root may itself be a link), so the CLI refuses the whole Bash-granting pass, read-only cases included (every run ends at 0 turns with a run error naming the credential store). On platform: darwin
with cli_version 2.1.293 or later, ignore links under ~/.docker/bin (Docker Desktop's own); an
older client, or a $DOCKER_CONFIG store elsewhere, keeps them not-ready. Write and Edit run
outside the shell sandbox and do not trigger it.
Refuse before any spend when sandbox_backend is not present and any case requests Bash,
PowerShell, Write, or Edit. Name both halves in the refusal: why each granting run would be
refused by the CLI (with no backend a run is refused rather than run unconfined) and score 0
rather than measuring anything, and the route.
absent or unknown: the backend is missing. On Linux or WSL2, install each missing package,
restart Claude Code, and confirm with /sandbox. Otherwise: WSL2, a Linux host with
bubblewrap and socat, macOS, or a Claude cloud session, whose host and missing packages come
from fetching What's available in cloud sessions (host) and Installed tools (packages) live and applying the matching table row.
not-ready: the backend is present and the packages resolve; the cause is the named links, and
the route is to resolve them or run on a host without them. Never suggest repointing
DOCKER_CONFIG to get past the check, which bypasses a sandbox safety check. On platform: wsl2, when DOCKER_CONFIG is unset (the swap changes only ~/.docker), every link (or ~/.docker itself) points under /mnt/, and no link target holds a tls/ directory (the script refuses one), run the CLI through the link swap without asking, and say so in the run report (<skill-dir> there is ${CLAUDE_SKILL_DIR}), so the run sees no links: sandboxes.md, Docker link swap. When any of the three fails, refuse and name the links.
Read-only suites run anywhere: Read, Glob, Grep, NotebookRead, Skill, AskUserQuestion,
Agent, TodoWrite, TaskCreate, TaskGet, TaskList, TaskUpdate, TaskStop (the
validator's READ_ONLY_TOOLS). suite_tools is read-only when every case's allowed_tools sits
inside that set; otherwise it lists the gated tools, which are exactly the ones needing an
--allow-tools grant. Neither a case nor a skill's own allowed-tools can widen that grant; an
ungranted tool is removed and reported on stderr as not granted.
Route per environment
| Environment |
Runs today, and the route |
Recorded run |
Pointer |
As of |
Recheck when |
| Windows native |
Read the pointer for whether this host has a backend, then apply the backend table: without one, a suite requesting Bash, PowerShell, Write or Edit takes the refusal above and its route, and one requesting none of them meets no refusal |
One read-only case passed on Claude Code 2.1.316; the CLI kept its unsealed sandbox, so remove each run's %TEMP%\claude-eval-* by hand as its warning says (record) |
https://code.claude.com/docs/en/plugin-evals#grant-tools |
2026-11-05 |
That section names a Windows backend, or a release note touches sandbox sealing on Windows |
| WSL2 |
Any suite once the backend is present, with Run step 3's TMPDIR. Docker Desktop's links make it not-ready: take that route's link swap |
Through the swap on Docker Desktop's own store, one Bash-granting case; also Bash-granting runs where a TMPDIR under /tmp or /var/tmp left the workspace read-only and one under home or /run/user/<uid> did not |
Set up Linux and WSL2; records.md |
2026-11-04 |
That section adds or drops a package or WSL note, or a release note touches sandbox credential exclusion |
| Claude cloud session |
Unverified: in the session, take the absent route's cloud clause, then the backend table |
None, by choice: the maintainers run no evals in cloud sessions, so whether the CLI runs there stays unverified (record) |
Installed tools |
2026-11-04 |
That page's "What's available in cloud sessions" section changes its host description, Installed tools changes its package list, or a run here is recorded |
| GitHub Actions |
In melodic-software/claude-code-plugins, dispatch eval-browser-dispatch.yml on main under the CI rules and keep its env-scrub input on, since off gives Bash subprocesses the Claude credentials. It passes no --threshold, so the job goes red below the CLI's default. Issue 6649 tracks the suite job. Elsewhere, write a job from CI and ci.md |
With TMPDIR moved out of /tmp (PR 12643), a Bash,Write case scored 1.00 in one run with env-scrub on, its graders matching a git merge call and the file that merge brings in (run). The earlier pass, also env-scrub on, logged only sandbox refusals under /tmp (record) |
Issue 6649; https://code.claude.com/docs/en/plugin-evals#run-evals-in-ci |
2026-11-04 |
Issue 6649 closes, that workflow's run step changes, or that section changes its runner requirements |
Worktree isolation
Under Claude Code's worktree isolation a Bash guard can refuse the CLI; which session shapes it
covers is in records. It matters only when run or init invokes
the CLI. Never retry, wrap, reroute, or otherwise bypass a refusal. Build the command with absolute
paths: for run, an absolute <target> and --json path inside the worktree; for init,
cd <absolute plugin root> && claude plugin eval init --bare <name>. Take the first route that fits,
and after a run, read the --json file back with the read action:
- A
run in a session entered with EnterWorktree: ask the user whether to leave the worktree
for it. Only on a yes, ExitWorktree with keep, run it, then EnterWorktree back by the
worktree's path; on a no, take route 3.
- A subagent: return the exact command to the caller, which takes these same routes.
- Otherwise print it for the user to run in a terminal outside Claude Code, or in a session
started with plain
claude (not --worktree) in the worktree.
Separately, the CLI refuses a plugin inside a repository with many linked worktrees (registers more linked worktrees than can be screened). Copy the plugin into a new empty directory outside
any git repository without git metadata or results (rsync -a --exclude .git --exclude 'evals/results' <plugin-dir>/ <new empty dir>/), run against the copy with an absolute target,
and point --json at the original's gitignored results directory.
Tested model and judge
Resolve the model under test and the judge from the planned invocation, the cases' model keys,
and the environment, by the --model and --judge-model rows of the command options table; an
alias and a full model ID naming the same model are the same. Report unknown when either cannot
be resolved before a run. Setting: ${user_config.same_model_warning}; if it renders empty or as
the literal placeholder text, use true, the manifest default, and say which you used. When it is
false, print same_model: off and compare nothing. When same_model is yes, print this line
under the report. It is advice: the run goes ahead.
warning: the judge is the model under test; pass a different --judge-model
Target routing
The plugin is the only unit the harness loads and the only thing the ablation measures. Every delta
is "this plugin versus no plugin".
| Target |
target_type |
Route |
A plugin root with plugin.json or .claude-plugin/plugin.json plus an eval dir |
plugin |
Direct. This is the designed path |
| A bare skill directory |
wrapped-skill |
Wrap it in a minimal .claude-plugin/plugin.json so it loads as <name>@skills-dir; claude plugin init <name> scaffolds that. Assert with a tool_used grader on tool: Skill with input_match naming the skill |
| An agent definition |
wrapped-agent |
Ship it in the wrapping plugin's agents, list Agent in the case's allowed_tools, assert tool_used on Agent. No first-party text describes this route; it composes documented parts, so treat a null result as a route defect before a plugin defect |
| Hooks and slash commands |
component of plugin |
Loaded as plugin components. Hooks run outside the sandbox, so a score for hooks you did not write is advisory unless the run is in a container or a CI runner. Nothing first-party grades command invocation as such |
| MCP servers |
component of plugin |
Mocked by default from evals/mocks/<server>/<tool>.md; a real server needs --allow-real-servers or --mocks off plus an --allow-tools "mcp__plugin_<plugin>_<server>__*" grant, and runs as you, outside the sandbox |
CLAUDE.md, .claude/rules, user settings, memory |
rules |
Refused. Every run starts from a throwaway home with user settings, hooks, CLAUDE.md, memory, MCP servers, and other plugins absent, so a shim that re-injects the rule measures the shim, not the rule |
A skill or agent that is not yet wrapped is not a target: report the wrap route, print it, and stop.
The next preflight sees wrapped-skill or wrapped-agent and proceeds.
For a rules target, invoke /harness-config:unhobble through the Skill tool when the
harness-config plugin is installed: it measures standing instructions by stripping them and
watching what the model stumbles over. When that plugin is absent, say so and describe the
experiment in one sentence (strip the instructions on a branch, log observed stumbles, restore only
what repeated evidence earns) so the user can run it by hand.
Cost
Each run and each judge grader is a real model call on the operator's account; there is no dry run.
agent runs = cases x runs x arms (arms = 2 under the default ablation, 1 under --ablation none)
judge calls = 3 per llm or baseline grader per run (the 2-of-3 vote)
Four grader types are free (regex, tool_used, tool_order, file_exists); llm and
baseline are billed. Carry the estimate as "roughly": the reported costUsd is a list-price
estimate, and arm costs are not symmetric.
A fresh suite is priced at 0.2 USD per run in either arm, judge calls included, and the figure is
called headroom. It has no pass of its own to scale, while cases x runs x arms is known before
any spend: four cases at three runs and two arms is 4.8 USD. Once a suite has run, scale from its
own last costUsd instead, estimating each arm from its own runs. Use 0.8 USD for a case's
without-arm only when a kept trace shows that arm loading a large skill; a suite with no such trace
is priced at 0.2 alone, and the older anchor is no caveat or risk to that estimate. An opus-judged
run costs more; size it from its own first run. When to re-derive the anchors: records.md.
Ceiling: ${user_config.max_cost_usd} USD, unlimited: ${user_config.unlimited_cost}. If either
renders empty or as the literal placeholder text, use 5 USD and false, the manifest defaults, and
state the ceiling you used without commenting on the setting's state.
- Print the estimate. This happens on every invocation, including unlimited, and the print is the
step that must appear in the transcript before any CLI call, not only in the final answer.
- Unlimited: drop
--max-cost-usd from the command and start. Never prompt; the estimate already
printed is the whole disclosure.
- Ceiling set and estimate under it: the pass goes through. Pass
--max-cost-usd <ceiling> and
start, without stopping to ask. A question about whether a pass fits gets the same answer: it
will go through and starts now under the ceiling, for example "about 4.8 USD, under the 5 USD
ceiling, so it starts with --max-cost-usd 5 and no confirmation".
- Ceiling set and estimate over it: stop and offer three exits, then proceed only on the answer:
raise the ceiling, narrow the run with
--case <glob> or --tag <tag>, or accept a partial run
knowing it exits 2 and its scores are not comparable.
The ceiling bounds a list-price estimate, not subscription usage, and it is checked before each run
launches, so an overrun is bounded by the runs already in flight (one, or up to --concurrency).
Cheapest levers first: deterministic graders only, then --ablation none (halves the runs and
loses the delta), then --case or --tag, then --runs 1 for iteration with a confirm at 3, then
a committed mocks/.replay/ so agent mocks replay without a model call.
Run
Run the static validator. It reaches no model:
python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>
Use python where python3 does not resolve. In this repository's own checkout
${CLAUDE_PLUGIN_ROOT} is plugins/evals, here and below. Exit 0 means proceed (WARN lines
are advice, not a stop); exit 1 means at least one FAIL, so stop and fix the cases before
spending, invoking the evals:validate skill through the Skill tool for the finding detail;
exit 2 means the eval dir is unreadable or the arguments are wrong, which is also a stop.
Run "Calibrating a judge" first unless every llm grader it can reproduce has passed with this
run's judge model. On a miss, fix the rubric or calibrate a stronger --judge-model and use it.
Invoke the CLI with the target first, ahead of every flag. Pointer for how the target is parsed:
https://code.claude.com/docs/en/plugin-evals#choose-what-to-evaluate, as of 2026-10-31; recheck when a release changes a flag that takes values.
claude plugin eval <target> --trust-plugin --keep-temp --judge-model opus --json results.json --threshold 0.8 --max-cost-usd <n> --no-publish
When preflight's suite_tools lists gated tools, add --allow-tools with exactly those tools;
without the grant each is removed from every run, so a case that must run or verify its own
work cannot. When a selected case's case.yaml sets context.scaffold_script, read each such
script first, since it runs as you outside the sandbox (see Gotchas), then add --scaffold.
Without it those cases run against an unstaged workspace, both arms score the same, and their
delta reads 0, so that share of the pass is wasted. When a script is one you would not run on
your own machine, select the other cases with --case <glob> or --tag <tag>. On Linux and WSL2, run the CLI as env TMPDIR=<dir> claude plugin eval …, where <dir> is a directory of at most 24 characters outside the system temp directories that you create first, on disk (for example $HOME/.ev when that stays within 24 characters; not the /run/user tmpfs, which kept sandboxes can fill). Why: sandboxes.md, Sandbox start. When a case's description asks for CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, add it to the same env prefix: sandboxes.md, Auto memory.
A non-TTY run against an untrusted directory needs --trust-plugin (Trust the plugin directory, as of 2026-11-10; recheck when that section changes what a run that cannot ask does).
--keep-temp keeps each run's trace for the validity gate under "Reading the delta". A run
that feeds a decision keeps --judge-model opus, since the CLI's default judge is uncalibrated
for decisions; only an exploratory run drops it. Run it in the foreground with a tool timeout
that covers the estimate (a three-case pass took about six minutes here) and wait. Stay
attached until every run exits: the results come back in the same tool call, so a headless
(-p) session runs suites one after another (-j adds concurrency inside one) and answers only
after all exit. A CLI backgrounded with a shell & gets no wait: the session answers before the
results exist and reads none of them. A command the main conversation starts with the Bash
tool's run_in_background holds a -p run open until it exits or reaches its time limit (both
in records). A larger suite can
outrun the foreground timeout or the time limit itself. For what holds a -p run open and which
limits apply, see https://code.claude.com/docs/en/headless#background-tasks-at-exit and
https://code.claude.com/docs/en/tools-reference#time-limit-for-background-commands. An
interactive session backgrounds a run only as a harness-tracked task it waits for. A run is
finished when the process exits and results.json exists; read the exit code and the JSON
together, never one alone. Write the file where git ignores it (the
CLI's own copy lands where the --output-dir row of Command options says, which can take --json too when ignored; as of 2026-11-10, recheck when that row changes its default); a run result is evidence to distill, not a file to commit.
Read the JSON with the read action below. With --json <file> there is no terminal table, so
the JSON is the only record of what happened.
Where a harness guard other than the worktree refusal above blocks a Bash command containing the
bare word eval, run the same command text, unchanged, through the PowerShell tool instead.
Reading the delta
Read in this order. Stopping early at any step is the finding.
Run the validity gate before reading any number:
python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/run-validity.py" results.json --runs <the run count the eval used> [--scaffolded]
The run count is the run's --runs, else the cases' runs, else 3. Add --scaffolded only when the eval ran with --scaffold (why). Without the traces
--keep-temp kept it reports the trace checks unchecked and the run INVALID. Report a score,
delta, or interval only from verdict: VALID (exit 0), naming any warnings it printed. On
verdict: INVALID (exit 1), report INVALID with every reason on that line and no number, then
fix the cause and rerun. Tell the user to post that INVALID line and its reasons in place of
the number; "post nothing" is not the instruction. Report a harness failure: reason as a
harness failure, never as the skill's result or a null delta, whatever the CLI's exit code, and
rerun by Sandbox start. Exit 2 means the file could not
be read or an argument was wrong: say which. The steps below still apply to a VALID run.
A with-arm denial aimed at or under the plugin's directory, at a directory above it (which covers
its files), or with no absolute path or no known plugin directory, is a FAIL: the agent reached
for a plugin file it could not read, so the fact belongs in the hub. So is a with-arm Bash call
whose result reports a sandbox network denial (Sources: sandbox network denial report), whose
hosts the summary names: the score measured a blocked request. Only an invocation probe, a
control, or a case whose prompt.md or case.yaml tags include no-network turns it into a warning; a
without-arm one is not read. The exceptions that turn a
denial into a warning (a denial the environment makes in both arms, such as a user-scope read of
~/.claude/standards; a denial in a case whose denial-free runs scored the same; an invocation
probe's denial; a sandbox network denial in an invocation probe, control or no-network case; a negative control's broken side, run with --own-copy-search-warns, searching its own copy, where such a case cannot decide the control; a finished without-arm run's search naming only the plugin it lacks), Bash denial paths, A/B gating,
the 0-turn context: fork run the gate passes, and invocation-probe misses and tools: reading-results.md, Denials.
partial. true (with partialReason of cost_ceiling, interrupted, or auth_failed) means
the suite did not finish: report that and keep the document out of any trend.
Per run, skippedPaidGraders: true or a non-null error. A skipped judge grader is still
scored, as a failure with explanation: "skipped: cost ceiling", so it silently depresses the
arm. A non-null error does not by itself imply score 0: which errored runs are still graded is in JSON result (as of 2026-11-10; recheck when that section changes which errored runs are graded).
Either makes the case not comparable; say so instead of reporting its number.
cases[].aggregates.delta. It is omitted when the arms are not comparable. An omitted delta
is never zero, and neither is a missing scoreWithout.
Only now read the delta: with-arm score minus without-arm score. A case the gate's ceiling
line names cannot show a gain: say it is excluded and use the delta that line gives over the
other cases.
Run the noise report over the same file and read its lines before calling any delta a gain:
python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/noise-report.py" results.json --threshold <the run's --threshold> --interval-method <method> --grader-agreement
<method> is ${user_config.interval_method}, and an empty or unfilled value there means the
default, normal, applies; do not mention the setting's state to the user. Pass --grader-agreement
unless ${user_config.grader_run_twice} is false (unexpanded, empty, or any other value means true,
the manifest default). Any other interval_method value is passed as is, and the script falls back to
normal. These values set the command this skill runs; they say nothing about a user's own run. Exit 2
means the file could not be read or an argument was malformed: say which, and report no interval.
Read the noise report's lines this way:
verdict: within noise or verdict: n too small to call: the gain is not established, whatever
the delta's sign or size. Say so before any number.
verdict: the interval excludes 0: report the delta together with its interval.
near ceiling: the baseline leaves no headroom, so the suite has almost no room to show a gain.
Add a case the model fails without the plugin before reading the delta again.
not comparable and score check: name the case. A score check means the run's reported score
and its graders disagree; read that run's graders before using its number.
judge agreement: a grader with split runs needs its explanation and evidence read before its
verdict is trusted. The line saying the file holds no judge votes means agreement is unknown,
not perfect.
cost: report it beside the scores, in the same answer as the delta.
- Two versions:
--baseline <before.json> --margin <m>; read compare verdict with each case drop.
pass count: the interval method (the interval_method setting, the --interval-method flag)
changes only this line, the count of cases at or above the threshold. Every score interval, the
delta line included, uses the normal method paired over cases whatever the setting, because a
case score is not a proportion of trials. So a normal delta interval under wilson is the
setting working as designed, and nothing needs checking (local-decisions.md, Interval method).
Spot-read outputs against verdicts before trusting any delta. For each case whose delta you
report, open a few runs from each arm (the grader explanations in results.json, and the kept
trace from --keep-temp), at least one its graders passed and one they failed where both exist,
and check each verdict against what the output does. A pass on a wrong output, or a fail on a
correct one, makes the grader or its rubric the finding: fix it and rerun before quoting the
number, and say which run showed it.
Check that every counted run did the task before its case's delta counts. A run that stopped early,
answered a different request or never reached the case's subject still gets a score, so averaging it
in moves the delta for a reason that is not the plugin. The test: each counted run's grader
explanations or kept trace show it attempted the task the prompt set and produced a result for it.
Name each run that did not, and report its case as not comparable, as step 3 does.
A gap holds for the model it ran on. A gap claimed for more than that model needs a run on a second
model (the same suite with a different --model), because one model's delta can come from that
model's habits rather than from the plugin's content. The test: the second run's noise report also
reads verdict: the interval excludes 0 with a delta of the same sign. Without that run, the report
names the one model the gap was measured on.
A case at 1.00 in both arms is a passing case and a null measurement at once. What else the number
means (with-only graders, arm: both, the ablation mode, a plugin directory named evals):
reading-results.md, What a delta does and does not say.
End the reading with one line on the change under test: promote only when the run is VALID, the
noise report's verdict is that the interval excludes 0 with the with-arm ahead, the spot-read found
no verdict it disagrees with, every counted run did the task, and a gap claimed beyond one model has
its second-model run; otherwise do not promote, followed by the first condition that failed. Then remove the run's sandboxes, once its traces are no longer needed: sandboxes.md.
A no-loss two-version compare ships a cut only after a detection control, the before text with the
changed rule removed, scores clearly lower on the same cases and model; otherwise report it as
unable to detect a loss: reading-results.md, Detection control.
Calibrating a judge
Trust an llm grader's scores only after its judge, the model the suite uses, agrees with labeled
answers on at least 90% of runs; a pass says nothing about another model. Before the first full
run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging
text); the default judge is uncalibrated until it passes. Report any other judge grader as
uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in the case's
samples/<grader>.json, set by three agents independently with the user settling every
disagreement:
python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" build --suite <eval-dir> --out <empty dir outside the repo>
claude plugin eval <out> --trust-plugin --ablation none --threshold 0 --runs 3 --judge-model <the suite's judge model> --keep-temp --no-publish --json <out>/results.json
python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" score --manifest <out>/manifest.json <out>/results.json
--threshold 0 keeps the CLI's exit code about errors, since must-fail cases are meant to score 0.
score prints a FAIL grader line for each grader under 90% and exits 1; fix that rubric, or move
the whole suite to a stronger judge, and calibrate again before reading its scores. Its false
positives and negatives name the samples to read first. A sample with no reproduced run is listed
untested, counts toward no agreement, and shows in the verdict line; raise --runs or tighten
the prompt before reading the grader's score. What build writes and skips, and which runs
score leaves out: case-authoring.md, Calibration samples.
CI
A CI job runs the command under Run with --model and --judge-model added, under three rules:
pin both, each to a full model ID and never an alias such as sonnet, so a model rollout is not
read as a plugin regression; pass --trust-plugin because a non-TTY job is otherwise refused; and
read the JSON in addition to the exit code. Exit 1 still fails the job on a below-threshold case,
but exit 1 has several causes and exit 2 more than one (the table in Run evals in CI lists them), so the JSON is what tells a
reader which one happened and whether the arms were comparable at all (pointer: the CI pins row under Sources).
Boundary, the built-in plugin eval command
- The CLI (
plugin eval, a built-in command) owns execution, grading, the ablation arms, the JSON and the HTML report. It is gated
by the version floor under Preflight, so this skill reads the version and never assumes the
command is available. This skill owns what surrounds a spend: the preflight, the no-spend
validation, the estimate and ceiling, target routing and the reading discipline. It never re-implements grading and never simulates a run or a score.
- Success criteria and case design belong to
/evals:design; come here once a suite exists.
- The skill-creator JSON eval format is not a suite this command reads. Convert its cases to case
directories (
/evals:design skill <name> writes them), check those with /evals:validate, then
run them here. To move one plugin's evals.json cases into its suite, run python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/convert-evals-json.py" <plugin-dir> (--dry-run prints the case count and writes nothing; --help prints what it writes), then /evals:validate on the plugin's evals/.
Next
- Validator FAIL, or a case file that failed to load:
/evals:validate <eval-dir>.
- Target turns out to be
CLAUDE.md or rules rather than a plugin: /harness-config:unhobble.
- Suite ran and the delta is read, and a case needs sharper criteria:
/evals:design <target>.
Gotchas
- A pass that crosses the ceiling late can still end
partial: false with exit 0 and one case
missing its delta: the ceiling skips judge calls, not runs. A pass that crosses it earlier
skips whole cases and reports partial: true with exit 2. Only the JSON distinguishes them.
- The without-arm is not the with-arm minus the plugin: the model without the plugin may load a
bundled skill such as
claude-api and cost several times the with-arm. Estimate each arm from
its own runs.
--trust-plugin persists (observed: records.md), so later non-TTY runs launch instead of being refused. How far a yes at the trust prompt reaches: Trust the plugin directory, as of 2026-11-10; recheck when that section changes what a yes trusts.
- Without
--keep-temp the per-run tracePath points at a directory the runner has already
deleted, so decide on --keep-temp before the run, not after reading a surprising score.
- A typo in
--case exits 1 and no exit code separates it from a real failure, so read the CLI's
message before treating an exit 1 as a failing case.
- A usage or rate limit mid-suite is not marked partial, so
partial: false does not show the
runs ended normally. Later runs end with the error and get 0 with no grader run, in both arms, so the suite reads as a regression (pointer: Runs fail with a usage-limit or rate-limit error partway through, as of 2026-11-10; recheck when that section changes how such a run is scored). Check each affected run's error
(cases[].arms.with[].error, cases[].arms.without[].error) before believing a drop; when it
names the limit, rerun those cases with --case once the limit resets. It is neither a
regression nor flaky cases, and leaving the cases out of the trend is not enough on its own.
When error is null, run the validity gate and the noise report before drawing any conclusion,
and say nothing yet about the cases themselves. A run whose sandbox never started fails the gate's session started check; the TMPDIR route is in sandboxes.md, Sandbox start. On Claude Code 2.1.316, when cases grant Bash or load a skill with a pre-computed (!`command` ) line, a run's TMPDIR must not sit under /tmp or /var/tmp; the run step's $HOME directory avoids it. The gate fails a run whose failed Bash result reports a read-only file system in the run's workspace or TMPDIR, or whose pre-computed command failed; recheck when a release note touches plugin eval sandbox paths or TMPDIR (records.md).
- Pass
--no-publish on a terminal run whose report must stay local. When a report publishes, and when a session-started run keeps it local: HTML report, as of 2026-11-10; recheck when that section changes when a report publishes.
- The sandbox limits what the a
…(truncated)
1---2name: plugin-eval3description: "Guided practice around `claude plugin eval`, which scores a plugin's eval suite: preflight, no-spend validation, a cost estimate, the run, its validity verdict, and the with-versus-without delta. Use when: planning how a headless `claude -p` session launches eval runs, an eval run's delta, arm scores or validity verdict is in question, including whether it can be shared (load this before answering), 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', missing socat or bubblewrap, 'how much will this eval cost', 'Bash refuses claude plugin eval', argument order around `--allow-tools`, 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format."4argument-hint: "[preflight|validate|run|read <json>|ci|init] [target]"5user-invocable: true6disable-model-invocation: false7metadata:8 source: https://skillmd.com/skills/melodic-software/plugin-eval9 workflow-stage: test10 summary: Preflight, price, run, and read a plugin eval suite around the CLI11---1213# Run a plugin eval suite1415`claude plugin eval` runs each case with the plugin and without it, grades each run, and writes16`aggregate-result.json`. This skill does what the CLI does not, all before money is spent: checks17that this machine can run the suite, prices the run, routes a target that is not a plugin, and18reads the delta. It never re-implements grading and never simulates a run or a score.1920## Actions2122| Action | What happens |23|---|---|24| `preflight <target>` | Print the report below and stop. No CLI invocation, no spend |25| `validate [<eval-dir>]` | Run the static case validator only. No model call |26| `run <target>` (default) | Preflight, validate, print the estimate, then invoke the CLI |27| `read <json>` | Read a written `aggregate-result.json` in the order that keeps a delta honest, then remove the sandboxes its run kept ([sandboxes.md](reference/sandboxes.md#removing-a-reads-sandboxes)) |28| `ci` | Answer from the CI section below |29| `init [<name>]` | Scaffold a suite: `claude plugin eval init --bare <name>` where there is no terminal |3031This page holds every rule the actions use; answer from it. Each upstream specific it states has a32row under [Sources](#sources). Background for a human reader:33[case-authoring.md](reference/case-authoring.md), [reading-results.md](reference/reading-results.md)34(JSON fields), [ci.md](reference/ci.md) (workflow file, exit codes, parser), and35[records.md](reference/records.md) (the observations behind each source row). Tuning one case36after a read, or confirming a change, is not an action and is answered from37[iterating.md](reference/iterating.md), never from this page alone.3839## Preflight4041Print every field, in order, before anything else (`leftover_sandboxes` and `tmp_free`: [probe](reference/sandboxes.md); `eval_tmpdir`: [Sandbox start](reference/sandboxes.md#sandbox-start)):4243```text44cli_version: <claude --version>45floor_met: <true | false>46git_version: <git --version | none>47platform: <windows | wsl2 | linux | darwin | other>48sandbox_backend: <present | absent | not-ready | unknown> (missing: <binaries that did not resolve> | links: <paths>)49target_type: <plugin | wrapped-skill | wrapped-agent | rules>50suite_tools: <read-only | the gated tools the cases request>51same_model: <yes | no | unknown | off> (tested <model>, judge <model>)52estimate_usd: roughly <n> (cases x runs x arms, plus judge calls)53ceiling: <n> USD | unlimited54leftover_sandboxes: <n> sandboxes (<total size> in <tmp>) | unknown55tmp_free: <free space on the temp dir's filesystem> | unknown56eval_tmpdir: <dir> (<n> characters) | too long: <dir> (<n> characters) | n/a57```5859`eval_tmpdir` is the `<dir>` the run step will pass as `TMPDIR` on Linux and WSL2, else `n/a`.60**Refuse before any spend** when it is longer than 24 characters, the longest length measured to61work ([Sandbox start](reference/sandboxes.md#sandbox-start)), and any case requests `Bash` or `PowerShell`; the route is a shorter `<dir>`, as in Run step 3.6263Preflight removes nothing and blocks on neither `leftover_sandboxes` nor `tmp_free`. It prints no removal command either: `<tmp>` can be shared with another session's run, whose live or kept sandboxes a glob over `<tmp>/claude-eval-*` would delete. When `leftover_sandboxes` is above zero, point at the removal by results file instead ([sandboxes.md](reference/sandboxes.md#leftover-sandboxes)). Report an unreadable path as `unknown`, never `0`.6465**Version floor:** Claude Code 2.1.269. Never assert that the command is installed: read `claude --version` and let the number decide. `floor_met: false` stops the run and reports the66floor; nothing else is worth doing on a binary that cannot execute a case. `plugin eval is67currently in early access` means an older binary: updating fixes it, no sign-up. `plugin eval is68currently unavailable` means Anthropic has switched the command off for now, possibly by account69or context: nothing local fixes it and there is no access to request, so wait and retry.7071**Git floor:** fetch <https://code.claude.com/docs/en/plugin-evals#requirements> live for it. A72git below it sets `floor_met: false` and names the version on the `git_version` line; no git73reports `git_version: none` and does not block.7475### Sandbox backend7677No CLI string reports the backend, so detection is platform-shaped. When the user states their78platform, answer for that platform and take `platform` from what they said. Never infer it from79this session's own host, which may not be the machine that will run the eval.8081| Observation | `sandbox_backend` |82|---|---|83| Native Windows (no WSL), or WSL1 (`wsl.exe -l -v` lists the distribution at `VERSION` 1) | `absent` |84| Linux, or WSL2 (`wsl.exe -l -v` lists the distribution at `VERSION` 2; `/proc/version` naming `microsoft` only says the host is WSL, since a custom WSL2 kernel can carry any release string), and every package the sandboxing page lists resolves (`command -v bwrap`, `command -v socat`) | `present` |85| Linux or WSL2, and any of them is missing | `absent`, naming each missing binary |86| macOS | `present` |87| Anything else | `unknown`, treated as `absent` for the refusal below |8889Then, when the table gives `present`, a case requests `Bash` or `PowerShell`, and90`find "${DOCKER_CONFIG:-$HOME/.docker}/" -mindepth 1 -type l` (read-only) prints a path, override91to `not-ready`, naming each path and the CLI's reason: the sandbox cannot reliably exclude a store92with a link inside it (the store's root may itself be a link), so the CLI refuses the whole `Bash`-granting pass, read-only cases included (every run ends at 0 turns with a run error naming the credential store). On `platform: darwin`93with `cli_version` 2.1.293 or later, ignore links under `~/.docker/bin` (Docker Desktop's own); an94older client, or a `$DOCKER_CONFIG` store elsewhere, keeps them `not-ready`. `Write` and `Edit` run95outside the shell sandbox and do not trigger it.9697**Refuse before any spend** when `sandbox_backend` is not `present` and any case requests `Bash`,98`PowerShell`, `Write`, or `Edit`. Name both halves in the refusal: why each granting run would be99refused by the CLI (with no backend a run is refused rather than run unconfined) and score 0100rather than measuring anything, and the route.101102- `absent` or `unknown`: the backend is missing. On Linux or WSL2, install each missing package,103 restart Claude Code, and confirm with `/sandbox`. Otherwise: WSL2, a Linux host with104 `bubblewrap` and `socat`, macOS, or a Claude cloud session, whose host and missing packages come105 from fetching [What's available in cloud sessions](https://code.claude.com/docs/en/cloud-environments#whats-available-in-cloud-sessions) (host) and [Installed tools](https://code.claude.com/docs/en/cloud-environments#installed-tools) (packages) live and applying the matching table row.106- `not-ready`: the backend is present and the packages resolve; the cause is the named links, and107 the route is to resolve them or run on a host without them. Never suggest repointing108 `DOCKER_CONFIG` to get past the check, which bypasses a sandbox safety check. On `platform: wsl2`, when `DOCKER_CONFIG` is unset (the swap changes only `~/.docker`), every link (or `~/.docker` itself) points under `/mnt/`, and no link target holds a `tls/` directory (the script refuses one), run the CLI through the link swap without asking, and say so in the run report (`<skill-dir>` there is `${CLAUDE_SKILL_DIR}`), so the run sees no links: [sandboxes.md, Docker link swap](reference/sandboxes.md#docker-link-swap). When any of the three fails, refuse and name the links.109110Read-only suites run anywhere: `Read`, `Glob`, `Grep`, `NotebookRead`, `Skill`, `AskUserQuestion`,111`Agent`, `TodoWrite`, `TaskCreate`, `TaskGet`, `TaskList`, `TaskUpdate`, `TaskStop` (the112validator's `READ_ONLY_TOOLS`). `suite_tools` is `read-only` when every case's `allowed_tools` sits113inside that set; otherwise it lists the gated tools, which are exactly the ones needing an114`--allow-tools` grant. Neither a case nor a skill's own `allowed-tools` can widen that grant; an115ungranted tool is removed and reported on stderr as `not granted`.116117### Route per environment118119| Environment | Runs today, and the route | Recorded run | Pointer | As of | Recheck when |120|---|---|---|---|---|---|121| Windows native | Read the pointer for whether this host has a backend, then apply the backend table: without one, a suite requesting `Bash`, `PowerShell`, `Write` or `Edit` takes the refusal above and its route, and one requesting none of them meets no refusal | One read-only case passed on Claude Code 2.1.316; the CLI kept its unsealed sandbox, so remove each run's `%TEMP%\claude-eval-*` by hand as its warning says ([record](https://github.com/melodic-software/claude-code-plugins/issues/8378#issuecomment-6662011078)) | <https://code.claude.com/docs/en/plugin-evals#grant-tools> | 2026-11-05 | That section names a Windows backend, or a release note touches sandbox sealing on Windows |122| WSL2 | Any suite once the backend is `present`, with Run step 3's `TMPDIR`. Docker Desktop's links make it `not-ready`: take that route's link swap | Through the swap on Docker Desktop's own store, one `Bash`-granting case; also `Bash`-granting runs where a `TMPDIR` under `/tmp` or `/var/tmp` left the workspace read-only and one under home or `/run/user/<uid>` did not | [Set up Linux and WSL2](https://code.claude.com/docs/en/sandboxing#set-up-linux-and-wsl2); [records.md](reference/records.md#sandbox-and-credential-store) | 2026-11-04 | That section adds or drops a package or WSL note, or a release note touches sandbox credential exclusion |123| Claude cloud session | Unverified: in the session, take the `absent` route's cloud clause, then the backend table | None, by choice: the maintainers run no evals in cloud sessions, so whether the CLI runs there stays unverified ([record](https://github.com/melodic-software/claude-code-plugins/issues/8378#issuecomment-6662011078)) | [Installed tools](https://code.claude.com/docs/en/cloud-environments#installed-tools) | 2026-11-04 | That page's "What's available in cloud sessions" section changes its host description, Installed tools changes its package list, or a run here is recorded |124| GitHub Actions | In melodic-software/claude-code-plugins, dispatch [`eval-browser-dispatch.yml`](https://github.com/melodic-software/claude-code-plugins/blob/main/.github/workflows/eval-browser-dispatch.yml) on main under the [CI](#ci) rules and keep its `env-scrub` input on, since off gives Bash subprocesses the Claude credentials. It passes no `--threshold`, so the job goes red below the [CLI's default](https://code.claude.com/docs/en/plugin-evals#command-options). [Issue 6649](https://github.com/melodic-software/claude-code-plugins/issues/6649) tracks the suite job. Elsewhere, write a job from [CI](#ci) and [ci.md](reference/ci.md) | With `TMPDIR` moved out of `/tmp` ([PR 12643](https://github.com/melodic-software/claude-code-plugins/pull/12643)), a `Bash,Write` case scored 1.00 in one run with `env-scrub` on, its graders matching a `git merge` call and the file that merge brings in ([run](https://github.com/melodic-software/claude-code-plugins/actions/runs/41706844559)). The earlier pass, also `env-scrub` on, logged only sandbox refusals under `/tmp` ([record](https://github.com/melodic-software/claude-code-plugins/issues/6649#issuecomment-6589725145)) | [Issue 6649](https://github.com/melodic-software/claude-code-plugins/issues/6649); <https://code.claude.com/docs/en/plugin-evals#run-evals-in-ci> | 2026-11-04 | Issue 6649 closes, that workflow's run step changes, or that section changes its runner requirements |125126### Worktree isolation127128Under Claude Code's worktree isolation a Bash guard can refuse the CLI; which session shapes it129covers is in [records](reference/records.md#worktrees). It matters only when `run` or `init` invokes130the CLI. Never retry, wrap, reroute, or otherwise bypass a refusal. Build the command with absolute131paths: for `run`, an absolute `<target>` and `--json` path inside the worktree; for `init`,132`cd <absolute plugin root> && claude plugin eval init --bare <name>`. Take the first route that fits,133and after a `run`, read the `--json` file back with the `read` action:1341351. A `run` in a session entered with `EnterWorktree`: ask the user whether to leave the worktree136 for it. Only on a yes, `ExitWorktree` with `keep`, run it, then `EnterWorktree` back by the137 worktree's path; on a no, take route 3.1382. A subagent: return the exact command to the caller, which takes these same routes.1393. Otherwise print it for the user to run in a terminal outside Claude Code, or in a session140 started with plain `claude` (not `--worktree`) in the worktree.141142Separately, the CLI refuses a plugin inside a repository with many linked worktrees (`registers143more linked worktrees than can be screened`). Copy the plugin into a new empty directory outside144any git repository without git metadata or results (`rsync -a --exclude .git --exclude145'evals/results' <plugin-dir>/ <new empty dir>/`), run against the copy with an absolute target,146and point `--json` at the original's gitignored results directory.147148### Tested model and judge149150Resolve the model under test and the judge from the planned invocation, the cases' `model` keys,151and the environment, by the `--model` and `--judge-model` rows of the command options table; an152alias and a full model ID naming the same model are the same. Report `unknown` when either cannot153be resolved before a run. Setting: `${user_config.same_model_warning}`; if it renders empty or as154the literal placeholder text, use `true`, the manifest default, and say which you used. When it is155`false`, print `same_model: off` and compare nothing. When `same_model` is `yes`, print this line156under the report. It is advice: the run goes ahead.157158```text159warning: the judge is the model under test; pass a different --judge-model160```161162## Target routing163164The plugin is the only unit the harness loads and the only thing the ablation measures. Every delta165is "this plugin versus no plugin".166167| Target | `target_type` | Route |168|---|---|---|169| A plugin root with `plugin.json` or `.claude-plugin/plugin.json` plus an eval dir | `plugin` | Direct. This is the designed path |170| A bare skill directory | `wrapped-skill` | Wrap it in a minimal `.claude-plugin/plugin.json` so it loads as `<name>@skills-dir`; `claude plugin init <name>` scaffolds that. Assert with a `tool_used` grader on `tool: Skill` with `input_match` naming the skill |171| An agent definition | `wrapped-agent` | Ship it in the wrapping plugin's `agents`, list `Agent` in the case's `allowed_tools`, assert `tool_used` on `Agent`. No first-party text describes this route; it composes documented parts, so treat a null result as a route defect before a plugin defect |172| Hooks and slash commands | component of `plugin` | Loaded as plugin components. Hooks run outside the sandbox, so a score for hooks you did not write is advisory unless the run is in a container or a CI runner. Nothing first-party grades command invocation as such |173| MCP servers | component of `plugin` | Mocked by default from `evals/mocks/<server>/<tool>.md`; a real server needs `--allow-real-servers` or `--mocks off` plus an `--allow-tools "mcp__plugin_<plugin>_<server>__*"` grant, and runs as you, outside the sandbox |174| `CLAUDE.md`, `.claude/rules`, user settings, memory | `rules` | **Refused.** Every run starts from a throwaway home with user settings, hooks, `CLAUDE.md`, memory, MCP servers, and other plugins absent, so a shim that re-injects the rule measures the shim, not the rule |175176A skill or agent that is not yet wrapped is not a target: report the wrap route, print it, and stop.177The next preflight sees `wrapped-skill` or `wrapped-agent` and proceeds.178179For a `rules` target, invoke `/harness-config:unhobble` through the Skill tool when the180`harness-config` plugin is installed: it measures standing instructions by stripping them and181watching what the model stumbles over. When that plugin is absent, say so and describe the182experiment in one sentence (strip the instructions on a branch, log observed stumbles, restore only183what repeated evidence earns) so the user can run it by hand.184185## Cost186187Each run and each judge grader is a real model call on the operator's account; there is no dry run.188189```text190agent runs = cases x runs x arms (arms = 2 under the default ablation, 1 under --ablation none)191judge calls = 3 per llm or baseline grader per run (the 2-of-3 vote)192```193194Four grader types are free (`regex`, `tool_used`, `tool_order`, `file_exists`); `llm` and195`baseline` are billed. Carry the estimate as "roughly": the reported `costUsd` is a list-price196estimate, and arm costs are not symmetric.197198**A fresh suite is priced at 0.2 USD per run in either arm, judge calls included, and the figure is199called headroom.** It has no pass of its own to scale, while cases x runs x arms is known before200any spend: four cases at three runs and two arms is 4.8 USD. Once a suite has run, scale from its201own last `costUsd` instead, estimating each arm from its own runs. Use 0.8 USD for a case's202without-arm only when a kept trace shows that arm loading a large skill; a suite with no such trace203is priced at 0.2 alone, and the older anchor is no caveat or risk to that estimate. An opus-judged204run costs more; size it from its own first run. When to re-derive the anchors: [records.md](reference/records.md#cost-anchors).205206Ceiling: `${user_config.max_cost_usd}` USD, unlimited: `${user_config.unlimited_cost}`. If either207renders empty or as the literal placeholder text, use 5 USD and `false`, the manifest defaults, and208state the ceiling you used without commenting on the setting's state.2092101. Print the estimate. This happens on every invocation, including unlimited, and the print is the211 step that must appear in the transcript before any CLI call, not only in the final answer.2122. Unlimited: drop `--max-cost-usd` from the command and start. Never prompt; the estimate already213 printed is the whole disclosure.2143. Ceiling set and estimate under it: the pass goes through. Pass `--max-cost-usd <ceiling>` and215 start, without stopping to ask. A question about whether a pass fits gets the same answer: it216 will go through and starts now under the ceiling, for example "about 4.8 USD, under the 5 USD217 ceiling, so it starts with `--max-cost-usd 5` and no confirmation".2184. Ceiling set and estimate over it: stop and offer three exits, then proceed only on the answer:219 raise the ceiling, narrow the run with `--case <glob>` or `--tag <tag>`, or accept a partial run220 knowing it exits 2 and its scores are not comparable.221222The ceiling bounds a list-price estimate, not subscription usage, and it is checked before each run223launches, so an overrun is bounded by the runs already in flight (one, or up to `--concurrency`).224Cheapest levers first: deterministic graders only, then `--ablation none` (halves the runs and225loses the delta), then `--case` or `--tag`, then `--runs 1` for iteration with a confirm at 3, then226a committed `mocks/.replay/` so agent mocks replay without a model call.227228## Run2292301. Run the static validator. It reaches no model:231232 ```bash233 python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>234 ```235236 Use `python` where `python3` does not resolve. In this repository's own checkout237 `${CLAUDE_PLUGIN_ROOT}` is `plugins/evals`, here and below. Exit 0 means proceed (WARN lines238 are advice, not a stop); exit 1 means at least one FAIL, so stop and fix the cases before239 spending, invoking the `evals:validate` skill through the Skill tool for the finding detail;240 exit 2 means the eval dir is unreadable or the arguments are wrong, which is also a stop.2412422. Run "Calibrating a judge" first unless every `llm` grader it can reproduce has passed with this243 run's judge model. On a miss, fix the rubric or calibrate a stronger `--judge-model` and use it.2442453. Invoke the CLI with the target first, ahead of every flag. Pointer for how the target is parsed:246 <https://code.claude.com/docs/en/plugin-evals#choose-what-to-evaluate>, as of 2026-10-31; recheck when a release changes a flag that takes values.247248 ```bash249 claude plugin eval <target> --trust-plugin --keep-temp --judge-model opus --json results.json --threshold 0.8 --max-cost-usd <n> --no-publish250 ```251252 When preflight's `suite_tools` lists gated tools, add `--allow-tools` with exactly those tools;253 without the grant each is removed from every run, so a case that must run or verify its own254 work cannot. When a selected case's `case.yaml` sets `context.scaffold_script`, read each such255 script first, since it runs as you outside the sandbox (see Gotchas), then add `--scaffold`.256 Without it those cases run against an unstaged workspace, both arms score the same, and their257 delta reads 0, so that share of the pass is wasted. When a script is one you would not run on258 your own machine, select the other cases with `--case <glob>` or `--tag <tag>`. On Linux and WSL2, run the CLI as `env TMPDIR=<dir> claude plugin eval …`, where `<dir>` is a directory of at most 24 characters outside the system temp directories that you create first, on disk (for example `$HOME/.ev` when that stays within 24 characters; not the `/run/user` tmpfs, which kept sandboxes can fill). Why: [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start). When a case's description asks for `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, add it to the same `env` prefix: [sandboxes.md, Auto memory](reference/sandboxes.md#auto-memory).259260 A non-TTY run against an untrusted directory needs `--trust-plugin` ([Trust the plugin directory](https://code.claude.com/docs/en/plugin-evals#trust-the-plugin-directory), as of 2026-11-10; recheck when that section changes what a run that cannot ask does).261 `--keep-temp` keeps each run's trace for the validity gate under "Reading the delta". A run262 that feeds a decision keeps `--judge-model opus`, since the CLI's default judge is uncalibrated263 for decisions; only an exploratory run drops it. Run it in the foreground with a tool timeout264 that covers the estimate (a three-case pass took about six minutes here) and wait. **Stay265 attached until every run exits:** the results come back in the same tool call, so a headless266 (`-p`) session runs suites one after another (`-j` adds concurrency inside one) and answers only267 after all exit. A CLI backgrounded with a shell `&` gets no wait: the session answers before the268 results exist and reads none of them. A command the main conversation starts with the Bash269 tool's `run_in_background` holds a `-p` run open until it exits or reaches its time limit (both270 in [records](reference/records.md#ceiling-trust-and-report-behavior)). A larger suite can271 outrun the foreground timeout or the time limit itself. For what holds a `-p` run open and which272 limits apply, see <https://code.claude.com/docs/en/headless#background-tasks-at-exit> and273 <https://code.claude.com/docs/en/tools-reference#time-limit-for-background-commands>. An274 interactive session backgrounds a run only as a harness-tracked task it waits for. A run is275 finished when the process exits and `results.json` exists; read the exit code and the JSON276 together, never one alone. Write the file where git ignores it (the277 CLI's own copy lands where the `--output-dir` row of [Command options](https://code.claude.com/docs/en/plugin-evals#command-options) says, which can take `--json` too when ignored; as of 2026-11-10, recheck when that row changes its default); a run result is evidence to distill, not a file to commit.2782794. Read the JSON with the `read` action below. With `--json <file>` there is no terminal table, so280 the JSON is the only record of what happened.281282Where a harness guard other than the worktree refusal above blocks a Bash command containing the283bare word `eval`, run the same command text, unchanged, through the PowerShell tool instead.284285## Reading the delta286287Read in this order. Stopping early at any step is the finding.2882891. Run the validity gate before reading any number:290291 ```bash292 python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/run-validity.py" results.json --runs <the run count the eval used> [--scaffolded]293 ```294295 The run count is the run's `--runs`, else the cases' `runs`, else 3. Add `--scaffolded` only when the eval ran with `--scaffold` ([why](reference/reading-results.md#read-order)). Without the traces296 `--keep-temp` kept it reports the trace checks unchecked and the run INVALID. Report a score,297 delta, or interval only from `verdict: VALID` (exit 0), naming any warnings it printed. On298 `verdict: INVALID` (exit 1), report INVALID with every reason on that line and no number, then299 fix the cause and rerun. Tell the user to post that INVALID line and its reasons in place of300 the number; "post nothing" is not the instruction. Report a `harness failure:` reason as a301 harness failure, never as the skill's result or a null delta, whatever the CLI's exit code, and302 rerun by [Sandbox start](reference/sandboxes.md#sandbox-start). Exit 2 means the file could not303 be read or an argument was wrong: say which. The steps below still apply to a VALID run.304305 A with-arm denial aimed at or under the plugin's directory, at a directory above it (which covers306 its files), or with no absolute path or no known plugin directory, is a FAIL: the agent reached307 for a plugin file it could not read, so the fact belongs in the hub. So is a with-arm Bash call308 whose result reports a sandbox network denial (Sources: sandbox network denial report), whose309 hosts the summary names: the score measured a blocked request. Only an invocation probe, a310 control, or a case whose `prompt.md` or `case.yaml` tags include `no-network` turns it into a warning; a311 without-arm one is not read. The exceptions that turn a312 denial into a warning (a denial the environment makes in both arms, such as a user-scope read of313 `~/.claude/standards`; a denial in a case whose denial-free runs scored the same; an invocation314 probe's denial; a sandbox network denial in an invocation probe, control or `no-network` case; a negative control's broken side, run with `--own-copy-search-warns`, searching its own copy, where such a case cannot decide the control; a finished without-arm run's search naming only the plugin it lacks), Bash denial paths, A/B gating,315 the 0-turn `context: fork` run the gate passes, and invocation-probe misses and tools: [reading-results.md, Denials](reference/reading-results.md#denials).3162. `partial`. `true` (with `partialReason` of `cost_ceiling`, `interrupted`, or `auth_failed`) means317 the suite did not finish: report that and keep the document out of any trend.3183. Per run, `skippedPaidGraders: true` or a non-null `error`. A skipped judge grader is still319 scored, as a failure with `explanation: "skipped: cost ceiling"`, so it silently depresses the320 arm. A non-null `error` does not by itself imply score 0: which errored runs are still graded is in [JSON result](https://code.claude.com/docs/en/plugin-evals#json-result) (as of 2026-11-10; recheck when that section changes which errored runs are graded).321 Either makes the case not comparable; say so instead of reporting its number.3224. `cases[].aggregates.delta`. It is **omitted** when the arms are not comparable. An omitted delta323 is never zero, and neither is a missing `scoreWithout`.3245. Only now read the delta: with-arm score minus without-arm score. A case the gate's `ceiling`325 line names cannot show a gain: say it is excluded and use the delta that line gives over the326 other cases.3276. Run the noise report over the same file and read its lines before calling any delta a gain:328329 ```bash330 python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/noise-report.py" results.json --threshold <the run's --threshold> --interval-method <method> --grader-agreement331 ```332333 `<method>` is `${user_config.interval_method}`, and an empty or unfilled value there means the334 default, `normal`, applies; do not mention the setting's state to the user. Pass `--grader-agreement`335 unless `${user_config.grader_run_twice}` is `false` (unexpanded, empty, or any other value means `true`,336 the manifest default). Any other `interval_method` value is passed as is, and the script falls back to337 `normal`. These values set the command this skill runs; they say nothing about a user's own run. Exit 2338 means the file could not be read or an argument was malformed: say which, and report no interval.339340Read the noise report's lines this way:341342- `verdict: within noise` or `verdict: n too small to call`: the gain is not established, whatever343 the delta's sign or size. Say so before any number.344- `verdict: the interval excludes 0`: report the delta together with its interval.345- `near ceiling`: the baseline leaves no headroom, so the suite has almost no room to show a gain.346 Add a case the model fails without the plugin before reading the delta again.347- `not comparable` and `score check`: name the case. A score check means the run's reported score348 and its graders disagree; read that run's graders before using its number.349- `judge agreement`: a grader with split runs needs its explanation and evidence read before its350 verdict is trusted. The line saying the file holds no judge votes means agreement is unknown,351 not perfect.352- `cost`: report it beside the scores, in the same answer as the delta.353- Two versions: `--baseline <before.json> --margin <m>`; read `compare verdict` with each `case drop`.354- `pass count`: the interval method (the `interval_method` setting, the `--interval-method` flag)355 changes only this line, the count of cases at or above the threshold. Every score interval, the356 delta line included, uses the normal method paired over cases whatever the setting, because a357 case score is not a proportion of trials. So a normal delta interval under `wilson` is the358 setting working as designed, and nothing needs checking ([local-decisions.md, Interval method](../methodology/reference/local-decisions.md#interval-method)).359360Spot-read outputs against verdicts before trusting any delta. For each case whose delta you361report, open a few runs from each arm (the grader explanations in `results.json`, and the kept362trace from `--keep-temp`), at least one its graders passed and one they failed where both exist,363and check each verdict against what the output does. A pass on a wrong output, or a fail on a364correct one, makes the grader or its rubric the finding: fix it and rerun before quoting the365number, and say which run showed it.366367Check that every counted run did the task before its case's delta counts. A run that stopped early,368answered a different request or never reached the case's subject still gets a score, so averaging it369in moves the delta for a reason that is not the plugin. The test: each counted run's grader370explanations or kept trace show it attempted the task the prompt set and produced a result for it.371Name each run that did not, and report its case as not comparable, as step 3 does.372373A gap holds for the model it ran on. A gap claimed for more than that model needs a run on a second374model (the same suite with a different `--model`), because one model's delta can come from that375model's habits rather than from the plugin's content. The test: the second run's noise report also376reads `verdict: the interval excludes 0` with a delta of the same sign. Without that run, the report377names the one model the gap was measured on.378379A case at 1.00 in both arms is a passing case and a null measurement at once. What else the number380means (with-only graders, `arm: both`, the ablation mode, a plugin directory named `evals`):381[reading-results.md, What a delta does and does not say](reference/reading-results.md#what-a-delta-does-and-does-not-say).382383End the reading with one line on the change under test: `promote` only when the run is VALID, the384noise report's verdict is that the interval excludes 0 with the with-arm ahead, the spot-read found385no verdict it disagrees with, every counted run did the task, and a gap claimed beyond one model has386its second-model run; otherwise `do not promote`, followed by the first condition that failed. Then remove the run's sandboxes, once its traces are no longer needed: [sandboxes.md](reference/sandboxes.md#removing-a-reads-sandboxes).387388A no-loss two-version compare ships a cut only after a detection control, the before text with the389changed rule removed, scores clearly lower on the same cases and model; otherwise report it as390unable to detect a loss: [reading-results.md, Detection control](reference/reading-results.md#detection-control).391392## Calibrating a judge393394Trust an `llm` grader's scores only after its judge, the model the suite uses, agrees with labeled395answers on at least 90% of runs; a pass says nothing about another model. Before the first full396run, calibrate every LLM-judge grader the harness can reproduce (one with labeled samples, judging397text); the default judge is uncalibrated until it passes. Report any other judge grader as398uncalibrated, never trusted silently. Labels are the must-pass and must-fail answers in the case's399`samples/<grader>.json`, set by three agents independently with the user settling every400disagreement:401402```bash403python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" build --suite <eval-dir> --out <empty dir outside the repo>404claude plugin eval <out> --trust-plugin --ablation none --threshold 0 --runs 3 --judge-model <the suite's judge model> --keep-temp --no-publish --json <out>/results.json405python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/calibrate-judge.py" score --manifest <out>/manifest.json <out>/results.json406```407408`--threshold 0` keeps the CLI's exit code about errors, since must-fail cases are meant to score 0.409410`score` prints a `FAIL grader` line for each grader under 90% and exits 1; fix that rubric, or move411the whole suite to a stronger judge, and calibrate again before reading its scores. Its false412positives and negatives name the samples to read first. A sample with no reproduced run is listed413`untested`, counts toward no agreement, and shows in the verdict line; raise `--runs` or tighten414the prompt before reading the grader's score. What `build` writes and skips, and which runs415`score` leaves out: [case-authoring.md, Calibration samples](reference/case-authoring.md#calibration-samples).416417## CI418419A CI job runs the command under Run with `--model` and `--judge-model` added, under three rules:420pin both, each to a full model ID and never an alias such as `sonnet`, so a model rollout is not421read as a plugin regression; pass `--trust-plugin` because a non-TTY job is otherwise refused; and422read the JSON in addition to the exit code. Exit 1 still fails the job on a below-threshold case,423but exit 1 has several causes and exit 2 more than one (the table in [Run evals in CI](https://code.claude.com/docs/en/plugin-evals#run-evals-in-ci) lists them), so the JSON is what tells a424reader which one happened and whether the arms were comparable at all (pointer: the CI pins row under [Sources](#sources)).425426## Boundary, the built-in `plugin eval` command427428- The **CLI** (`plugin eval`, a built-in command) owns execution, grading, the ablation arms, the JSON and the HTML report. It is gated429 by the version floor under Preflight, so this skill reads the version and never assumes the430 command is available. This skill owns what surrounds a spend: the preflight, the no-spend431 validation, the estimate and ceiling, target routing and the reading discipline. It never re-implements grading and never simulates a run or a score.432- Success criteria and case design belong to `/evals:design`; come here once a suite exists.433- The skill-creator JSON eval format is not a suite this command reads. Convert its cases to case434 directories (`/evals:design skill <name>` writes them), check those with `/evals:validate`, then435 run them here. To move one plugin's `evals.json` cases into its suite, run `python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/convert-evals-json.py" <plugin-dir>` (`--dry-run` prints the case count and writes nothing; `--help` prints what it writes), then `/evals:validate` on the plugin's `evals/`.436437## Next438439- Validator FAIL, or a case file that failed to load: `/evals:validate <eval-dir>`.440- Target turns out to be `CLAUDE.md` or rules rather than a plugin: `/harness-config:unhobble`.441- Suite ran and the delta is read, and a case needs sharper criteria: `/evals:design <target>`.442443## Gotchas444445- A pass that crosses the ceiling late can still end `partial: false` with exit 0 and one case446 missing its `delta`: the ceiling skips judge calls, not runs. A pass that crosses it earlier447 skips whole cases and reports `partial: true` with exit 2. Only the JSON distinguishes them.448- The without-arm is not the with-arm minus the plugin: the model without the plugin may load a449 bundled skill such as `claude-api` and cost several times the with-arm. Estimate each arm from450 its own runs.451- `--trust-plugin` persists (observed: [records.md](reference/records.md#ceiling-trust-and-report-behavior)), so later non-TTY runs launch instead of being refused. How far a yes at the trust prompt reaches: [Trust the plugin directory](https://code.claude.com/docs/en/plugin-evals#trust-the-plugin-directory), as of 2026-11-10; recheck when that section changes what a yes trusts.452- Without `--keep-temp` the per-run `tracePath` points at a directory the runner has already453 deleted, so decide on `--keep-temp` before the run, not after reading a surprising score.454- A typo in `--case` exits 1 and no exit code separates it from a real failure, so read the CLI's455 message before treating an exit 1 as a failing case.456- A usage or rate limit mid-suite is **not** marked partial, so `partial: false` does not show the457 runs ended normally. Later runs end with the error and get 0 with no grader run, in both arms, so the suite reads as a regression (pointer: [Runs fail with a usage-limit or rate-limit error partway through](https://code.claude.com/docs/en/plugin-evals#runs-fail-with-a-usage-limit-or-rate-limit-error-partway-through), as of 2026-11-10; recheck when that section changes how such a run is scored). Check each affected run's `error`458 (`cases[].arms.with[].error`, `cases[].arms.without[].error`) before believing a drop; when it459 names the limit, rerun those cases with `--case` once the limit resets. It is neither a460 regression nor flaky cases, and leaving the cases out of the trend is not enough on its own.461 When `error` is null, run the validity gate and the noise report before drawing any conclusion,462 and say nothing yet about the cases themselves. A run whose sandbox never started fails the gate's `session started` check; the `TMPDIR` route is in [sandboxes.md, Sandbox start](reference/sandboxes.md#sandbox-start). On Claude Code 2.1.316, when cases grant Bash or load a skill with a pre-computed (``!`command` ``) line, a run's `TMPDIR` must not sit under `/tmp` or `/var/tmp`; the run step's `$HOME` directory avoids it. The gate fails a run whose failed Bash result reports a read-only file system in the run's workspace or `TMPDIR`, or whose pre-computed command failed; recheck when a release note touches `plugin eval` sandbox paths or `TMPDIR` ([records.md](reference/records.md#sandbox-and-credential-store)).463- Pass `--no-publish` on a terminal run whose report must stay local. When a report publishes, and when a session-started run keeps it local: [HTML report](https://code.claude.com/docs/en/plugin-evals#html-report), as of 2026-11-10; recheck when that section changes when a report publishes.464- The sandbox limits what the a465466…(truncated)