eval-kit
claude plugin eval runs a case against your plugin and, by default, against a baseline with the plugin absent. The number worth reading is the difference between the two arms, and it only means something if the plugin actually loaded in the with arm.
Runs spend money. This plugin ships scaffolds only. No eval from this suite has been run, so there is no figure from them yet. Never start one without an explicit cost cap and the person's yes.
1. Write the case
Copy ${CLAUDE_PLUGIN_ROOT}/templates/case.yaml into the plugin's evals/<case-name>/case.yaml and edit it. (claude plugin eval init --bare <name> writes a different, blank prompt-and-graders layout; without --bare it starts an interview and needs an interactive terminal.) Fields:
- Top level:
schema_version (quoted, "1.0"), name, description, tags, runs (at most 50), graders (at least one), expected_outcome. Optional: plugins, context (add_dirs, scaffold_script, history_file).
execution: prompt, max_turns (at most 200), timeout_seconds (at most 3600), model, allowed_tools; also append_system_prompt, env, artifact_publish. Set the timeout generously: a run that times out is graded on what it finished, which is often nothing.
- Graders, each with a unique
name, an optional weight, and arm. Unknown keys are rejected.
tool_used: tool, input_match, min, max.
tool_order: before, after.
regex: target (last_message, trace, files, mock_calls, or {source: file, path: <relative path>}; a bare path string is rejected), pattern, flags, match (contains, not_contains, count:N).
file_exists: path, exists.
llm: criteria, focus (takes the same values as a regex target).
baseline: baseline_file, criteria.
arm: with-only marks a grader as a fired-indicator: it shows whether the plugin did something and adds nothing to the score. A tool_used grader on Skill is display-only by default under with/without, so a must-not-fire check on a skill needs min: 0, max: 0, arm: both.
claude plugin validate does not read case.yaml. A clean validate says nothing about your cases; a broken one fails only when someone pays to run it. Check keys against the list above.
Write the prompt so it stands alone: code inline, no absolute paths, no home-directory paths, since cases run in a sandbox directory. Someone other than the plugin's author should write the prompts. Include at least one case where the plugin should not fire, with a grader that rewards it staying out.
2. Pilot gate
claude plugin eval --runs 1 --ablation with-without --no-scaffold --no-publish --max-cost-usd <cap> \
--allow-tools Write Edit "Bash(python3:*)" --judge-model opus <plugin-path>
Tools follow graders: Bash, Write and Edit (also WebFetch and mcp__*) are gated, and a run only gets them through --allow-tools. Grant exactly what the prompt needs, scoped as above, and list in allowed_tools only tools a grader or the task uses. Without the grant the with arm can't do the task and the delta means nothing.
--no-publish is required because publishing to claude.ai is the default. Then:
- Open
<evals>/results/*/aggregate-result.json and check suite.plugins lists your plugin with an empty problems list (the codes include manifest_invalid, disabled_by_default, will_not_load, identity_unverified and archive_not_probed; any entry counts). If one is there, the with arm ran without the plugin and any delta means nothing. Fix that first.
- Read the top-level
costUsd, multiply by the number of runs you plan, tell the person the figure, and get their go.
- Set
--judge-model opus for llm graders; the default judge is haiku. The judge must be a different, larger model than execution.model, because a model favors its own output. Give an llm grader that judges ordering focus: trace; last_message is the default and can't see earlier turns.
--trust-plugin only on an explicit yes.
Exit codes: with the default --threshold 1.0, exit 1 is normal when scores are partial; it isn't a crash. --max-cost-usd exits 2 when it trips. Useful flags: --runs, --case, --tag, -j, --report.
3. Read the delta
The headline is the with-plugin score minus the without-plugin score, per case.
- Use three or more runs per case before you read anything into a gap. One run is a pilot.
- Compare against the spread between repeat runs of the same arm. A gap inside that spread isn't a finding.
- Look at the sign across cases before the size. Consistent direction in many cases beats one big number.
- Read the case where the plugin should lose first. If the with arm wins there, suspect the grader.
- A grader written by the same person who wrote the plugin tends to reward what the plugin does. Check that each score-bearing grader would fail on a run that's actually bad.
- State what you ran: runs, cap, model, date, tool version. Report a figure only with that next to it.
Scaffolds in this plugin
${CLAUDE_PLUGIN_ROOT}/evals/protocol-should-lose/case.yaml and ${CLAUDE_PLUGIN_ROOT}/evals/with-without-delta/case.yaml. Both are tagged scaffold and not-run.
1---2name: eval-kit3description: "Use when scaffolding or reading claude plugin eval runs: write case.yaml, pilot one run under a cost cap, confirm the plugin loaded, then read the with-minus-without delta."4model: inherit5category: learn-research6metadata:7 source: https://skillmd.com/skills/dojogenesis/eval-kit8---910# eval-kit1112`claude plugin eval` runs a case against your plugin and, by default, against a baseline with the plugin absent. The number worth reading is the difference between the two arms, and it only means something if the plugin actually loaded in the with arm.1314**Runs spend money.** This plugin ships scaffolds only. No eval from this suite has been run, so there is no figure from them yet. Never start one without an explicit cost cap and the person's yes.1516## 1. Write the case1718Copy `${CLAUDE_PLUGIN_ROOT}/templates/case.yaml` into the plugin's `evals/<case-name>/case.yaml` and edit it. (`claude plugin eval init --bare <name>` writes a different, blank prompt-and-graders layout; without `--bare` it starts an interview and needs an interactive terminal.) Fields:1920- Top level: `schema_version` (quoted, `"1.0"`), `name`, `description`, `tags`, `runs` (at most 50), `graders` (at least one), `expected_outcome`. Optional: `plugins`, `context` (`add_dirs`, `scaffold_script`, `history_file`).21- `execution`: `prompt`, `max_turns` (at most 200), `timeout_seconds` (at most 3600), `model`, `allowed_tools`; also `append_system_prompt`, `env`, `artifact_publish`. Set the timeout generously: a run that times out is graded on what it finished, which is often nothing.22- Graders, each with a unique `name`, an optional `weight`, and `arm`. Unknown keys are rejected.23 - `tool_used`: `tool`, `input_match`, `min`, `max`.24 - `tool_order`: `before`, `after`.25 - `regex`: `target` (`last_message`, `trace`, `files`, `mock_calls`, or `{source: file, path: <relative path>}`; a bare path string is rejected), `pattern`, `flags`, `match` (`contains`, `not_contains`, `count:N`).26 - `file_exists`: `path`, `exists`.27 - `llm`: `criteria`, `focus` (takes the same values as a regex `target`).28 - `baseline`: `baseline_file`, `criteria`.29- `arm: with-only` marks a grader as a fired-indicator: it shows whether the plugin did something and adds nothing to the score. A `tool_used` grader on `Skill` is display-only by default under with/without, so a must-not-fire check on a skill needs `min: 0`, `max: 0`, `arm: both`.3031`claude plugin validate` does not read case.yaml. A clean validate says nothing about your cases; a broken one fails only when someone pays to run it. Check keys against the list above.3233Write the prompt so it stands alone: code inline, no absolute paths, no home-directory paths, since cases run in a sandbox directory. Someone other than the plugin's author should write the prompts. Include at least one case where the plugin should not fire, with a grader that rewards it staying out.3435## 2. Pilot gate3637```38claude plugin eval --runs 1 --ablation with-without --no-scaffold --no-publish --max-cost-usd <cap> \39 --allow-tools Write Edit "Bash(python3:*)" --judge-model opus <plugin-path>40```4142Tools follow graders: `Bash`, `Write` and `Edit` (also `WebFetch` and `mcp__*`) are gated, and a run only gets them through `--allow-tools`. Grant exactly what the prompt needs, scoped as above, and list in `allowed_tools` only tools a grader or the task uses. Without the grant the with arm can't do the task and the delta means nothing.4344`--no-publish` is required because publishing to claude.ai is the default. Then:45461. Open `<evals>/results/*/aggregate-result.json` and check `suite.plugins` lists your plugin with an empty problems list (the codes include `manifest_invalid`, `disabled_by_default`, `will_not_load`, `identity_unverified` and `archive_not_probed`; any entry counts). If one is there, the with arm ran without the plugin and any delta means nothing. Fix that first.472. Read the top-level `costUsd`, multiply by the number of runs you plan, tell the person the figure, and get their go.483. Set `--judge-model opus` for `llm` graders; the default judge is haiku. The judge must be a different, larger model than `execution.model`, because a model favors its own output. Give an `llm` grader that judges ordering `focus: trace`; `last_message` is the default and can't see earlier turns.494. `--trust-plugin` only on an explicit yes.5051Exit codes: with the default `--threshold 1.0`, exit 1 is normal when scores are partial; it isn't a crash. `--max-cost-usd` exits 2 when it trips. Useful flags: `--runs`, `--case`, `--tag`, `-j`, `--report`.5253## 3. Read the delta5455The headline is the with-plugin score minus the without-plugin score, per case.5657- Use three or more runs per case before you read anything into a gap. One run is a pilot.58- Compare against the spread between repeat runs of the same arm. A gap inside that spread isn't a finding.59- Look at the sign across cases before the size. Consistent direction in many cases beats one big number.60- Read the case where the plugin should lose first. If the with arm wins there, suspect the grader.61- A grader written by the same person who wrote the plugin tends to reward what the plugin does. Check that each score-bearing grader would fail on a run that's actually bad.62- State what you ran: runs, cap, model, date, tool version. Report a figure only with that next to it.6364## Scaffolds in this plugin6566`${CLAUDE_PLUGIN_ROOT}/evals/protocol-should-lose/case.yaml` and `${CLAUDE_PLUGIN_ROOT}/evals/with-without-delta/case.yaml`. Both are tagged `scaffold` and `not-run`.