skill-eval
A skill can read well and still add nothing — the model would have done the
same work without it. claude plugin eval (Claude Code ≥ 2.1.269) measures
that directly: every case runs in an isolated session with the skill loaded
and again without it, and the report's Δ (delta) is the difference.
Δ > 0 is the skill's contribution. Δ ≈ 0 on a case the skill claims to own
means it isn't pulling its weight.
This skill wraps that runner for single skills in this repo. Its job is to
make suites correct and keep spend near zero: cheap checks are free, and the
only step that runs Claude sessions demands a cap you chose.
What "cost" means here. Every case run is a full Claude Code session on
whatever credential the CLI is logged in with. Logged in through claude.ai
(claude auth status shows authMethod: claude.ai), runs draw on the
subscription's usage allowance, not dollars. The runner's costUsd and
--max-cost-usd are estimated API-equivalent figures, which still work as a
size gauge and a cap. Real dollars are spent only with an API key, or with
paid extra usage past the plan limit. Check claude auth status before
telling the owner what a run will cost them.
Where this fits
| Question |
Skill |
| Which skill fires for a request; who steals whose triggers |
skill-triage (Jev, fractions of a cent) |
| Does the skill help once it fires — measured, with vs without |
skill-eval |
| Is the SKILL.md well written, within budget, spec-conformant |
skill-review |
| Draft, iterate and hand-review outputs while authoring |
skill-creator (its evals/evals.json) |
evals/evals.json (skill-creator's authoring format) and plugin-eval case
directories live side by side in the same evals/ folder. convert turns the
first into the second; it never deletes or rewrites either.
The script
scripts/skill_eval.py (Python stdlib). Run it from the repo root with a
skill directory as the target:
| Mode |
Cost |
What it does |
convert <skill> [--negative "…"] |
free |
Each evals.json entry → a case dir with prompt.md, a display-only "skill fired" check and a judge rubric from its assertions. --negative adds a must-not-fire case. Existing case dirs are kept untouched. |
lint <skill>… |
free |
Enforces the runner's floor rules (below), flags sandbox-breaking paths, and prints the suite's session count and paid-grader count. |
lint --all skills/ |
free |
The same checks for every skill that has plugin-eval cases. CI runs this on every PR. |
run <skill> --max-cost-usd N |
usage |
Lints, then runs claude plugin eval with/without ablation and a local-only report. The cap is required; there is no default. |
report <skill> |
free |
Δ, scores and the real costUsd from the latest evals/results/*/aggregate-result.json. |
Exit codes: 0 ok, 1 lint failure or a case below --threshold, 2 argument
error, 3 claude CLI missing or too old, 5 stopped at the cap (partial
results kept). Results land in <skill>/evals/results/, which is gitignored.
The floor rules (why lint fails a suite)
These come from the runner's own authoring guide. Each one exists because
without it the number stops meaning anything:
- At least one must-not-fire case. A suite with only positive cases can't
tell a skill that helps from one that fires on everything. The check is a
tool_used grader with tool: Skill, min: 0, max: 0, and
arm: both — leave out arm and it becomes display-only and never scores.
- Every case has an outcome grader, not just "the skill fired". A plain
fired-check is excluded from the score under ablation; it can't move Δ.
Grade what the user would notice: the answer, the file, the format.
runs ≥ 3 in the case file. Single runs are noise. For a pilot, pass
--runs 1 on the command line instead of lowering the floor in the file.
- No absolute or
~/ paths — cases run in a sandbox working directory.
- Set
timeout_seconds and allowed_tools to fit the work. An
under-set timeout scores 0, not "timed out", which reads as a broken skill.
Tools beyond Read/Glob/Grep/Skill (Bash, Write, Edit, WebFetch, mcp__*)
also need an operator grant at run time: --allow-tools Write.
Cases that can actually measure something
A case can follow every floor rule and still be unable to show a difference.
The common ways that happens, all seen in real conversions:
- The input isn't in the sandbox. evals.json prompts often point at
~/repos/x or an "attached" doc. In the sandbox both arms just report "not
found", and Δ ≈ 0 says nothing about the skill. Build a small fixture repo
and copy it in with a case.yaml scaffold_script, then run with
--scaffold. Put contradictions in the fixture (a doc claim the code
disproves) so a grader can tell careful work from a skim.
- The skill pauses to ask. Eval sessions can't answer a question. If the
skill confirms before acting (an output folder, a destructive step), state
the answer in the prompt ("write to
./out/ — confirmed"). Otherwise the
with-arm scores worse for following its own rule.
- Writing skills get read-only defaults.
convert gives every case
Read/Glob/Grep/Skill, 20 turns and 600 s, and its notes flag cases that look
like they write files. Those need Write in allowed_tools, a real budget
(deep-dive-sized work can take 60 turns and 1500 s), --allow-tools Write
at run time, and graders that read the file (regex with
target: {source: file, path}, or file_exists). A judge on the last
message can't see what the skill wrote.
--ablation none isn't a cheaper Δ. It drops the without-arm, so there
is no Δ at all. To save money, pilot one case with --runs 1.
Grader choice is a cost choice
| Type |
Cost |
Use for |
regex |
free |
Mechanical claims: a ```mermaid block exists, a heading is present, a phrase is absent (match: not_contains) |
file_exists |
free |
The skill must create a file (path: <glob>) |
tool_used / tool_order |
free |
Fired / must-not-fire; tool sequencing |
llm |
paid judge |
Claims a pattern can't check. Write the rubric as concrete checkable claims |
baseline |
paid judge |
Preference between arms — rarely worth it |
convert writes assertions into one llm rubric because evals.json
assertions are prose. After converting, rewrite any assertion that's really a
pattern as a free regex grader. The judge defaults to Haiku; the runner's
guide recommends --judge-model sonnet for nuanced rubrics, which costs more.
That trade is the owner's to make.
The judge is the small cost. Sessions are the big one: a full run is
(cases × runs × 2 arms) complete Claude sessions, each doing real work. lint
prints that count so the size is visible before anything is spent.
The spend protocol
Session runs follow the same rule as every paid call in this repo: estimate,
run the smallest version, never raise a cap without asking.
lint — free. Read the session count.
- Pilot one case:
run <skill> --case '<one>' --runs 1 --max-cost-usd 0.25.
Use the must-not-fire case first; it's the shortest.
report — real costUsd for those 2 sessions. Multiply by the session
count from step 1 for the full suite, and tell the owner the figure
(subscription usage, or dollars on an API key) before running it.
- Full run only on the owner's yes, with a cap at that figure plus headroom.
Exit 5 means the cap stopped it; report the partial result. Don't rerun
with a higher cap on your own.
Trust. The first run in a folder asks whether you trust the plugin; a
non-interactive run can't answer. Ask the owner once, and pass --trust only
after a yes. It doesn't grant extra tools or scaffold scripts.
Publishing. run keeps the HTML report local (--no-publish). Pass
--publish only when the owner wants it on claude.ai.
Reading a result
- Δ > 0 on the skill's own cases — it helps. Keep it.
- Δ ≈ 0 where the skill claims value — the model already does this
unaided. That's a prune or rewrite candidate for
skill-update, or a sign
the cases are too easy (the model only reaches for skills on tasks it can't
do trivially, so trivial prompts score the same with or without).
- Must-not-fire case fails — the description over-triggers. Take the
prompt to
skill-triage collisions --skill <name> to see who it's stealing
from.
report warns a plugin problem (manifest_invalid,
disabled_by_default, will_not_load) — the with-arm ran without the skill.
The scores are meaningless; fix loading first.
- An implausible jump between runs — suspect the judge being gamed;
spot-check the transcripts before believing it.
Re-run a skill's suite after a model or Claude Code upgrade. Harness
behaviour shifts, and model-adaptation pruning should be backed by a Δ, not
a hunch.
Anti-patterns
| Anti-pattern |
Why it fails |
| Running the full suite before a one-case pilot |
You learn the cost after paying it |
| A suite with no must-not-fire case |
Can't tell a useful skill from an over-eager one |
tool_used: Skill as a case's only grader |
Excluded from the score; Δ is always 0 |
llm graders for mechanical checks |
Paying a judge for what a regex does free |
Lowering runs in the case file to save money |
Breaks the floor; use --runs 1 for pilots only |
Raising --max-cost-usd after exit 5 without asking |
The cap is the owner's decision |
| Absolute paths in prompts |
The sandbox has a different cwd; the case fails for the wrong reason |
1---2name: skill-eval3version: 0.1.24description: >-5 Measure whether a skill actually helps: build a `claude plugin eval` suite,6 run it with and without the skill under a required spend cap, and read the7 delta. Use when adding evals to a skill, converting evals.json, re-checking8 skills after a model upgrade, or deciding if a skill should be cut. Trigger9 on: "plugin eval", "eval this skill", "does this skill help", "skill delta".10requires_agent_teams: false11requires_claude_code: true12min_plan: starter13owns:14 directories: []15 patterns: []16 shared_read: ["skills/"]17allowed-tools: ["Read", "Write", "Bash", "Grep", "Glob"]18composes_with:19 [20 "skill-triage",21 "skill-review",22 "skill-update",23 "skill-writer",24 "model-adaptation",25 ]26spawned_by: []27metadata:28 source: https://skillmd.com/skills/ivy00johns/skill-eval29---3031# skill-eval3233A skill can read well and still add nothing — the model would have done the34same work without it. `claude plugin eval` (Claude Code ≥ 2.1.269) measures35that directly: every case runs in an isolated session **with** the skill loaded36and again **without** it, and the report's **Δ (delta)** is the difference.37Δ > 0 is the skill's contribution. Δ ≈ 0 on a case the skill claims to own38means it isn't pulling its weight.3940This skill wraps that runner for single skills in this repo. Its job is to41make suites correct and keep spend near zero: cheap checks are free, and the42only step that runs Claude sessions demands a cap you chose.4344**What "cost" means here.** Every case run is a full Claude Code session on45whatever credential the CLI is logged in with. Logged in through claude.ai46(`claude auth status` shows `authMethod: claude.ai`), runs draw on the47subscription's usage allowance, not dollars. The runner's `costUsd` and48`--max-cost-usd` are estimated API-equivalent figures, which still work as a49size gauge and a cap. Real dollars are spent only with an API key, or with50paid extra usage past the plan limit. Check `claude auth status` before51telling the owner what a run will cost them.5253## Where this fits5455| Question | Skill |56| --------------------------------------------------------------- | ----------------------------------------- |57| Which skill _fires_ for a request; who steals whose triggers | `skill-triage` (Jev, fractions of a cent) |58| Does the skill _help_ once it fires — measured, with vs without | **skill-eval** |59| Is the SKILL.md well written, within budget, spec-conformant | `skill-review` |60| Draft, iterate and hand-review outputs while authoring | `skill-creator` (its `evals/evals.json`) |6162`evals/evals.json` (skill-creator's authoring format) and plugin-eval case63directories live side by side in the same `evals/` folder. `convert` turns the64first into the second; it never deletes or rewrites either.6566## The script6768`scripts/skill_eval.py` (Python stdlib). Run it from the repo root with a69skill directory as the target:7071| Mode | Cost | What it does |72| ---------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |73| `convert <skill> [--negative "…"]` | free | Each `evals.json` entry → a case dir with `prompt.md`, a display-only "skill fired" check and a judge rubric from its assertions. `--negative` adds a must-not-fire case. Existing case dirs are kept untouched. |74| `lint <skill>…` | free | Enforces the runner's floor rules (below), flags sandbox-breaking paths, and prints the suite's session count and paid-grader count. |75| `lint --all skills/` | free | The same checks for every skill that has plugin-eval cases. CI runs this on every PR. |76| `run <skill> --max-cost-usd N` | **usage** | Lints, then runs `claude plugin eval` with/without ablation and a local-only report. The cap is required; there is no default. |77| `report <skill>` | free | Δ, scores and the real `costUsd` from the latest `evals/results/*/aggregate-result.json`. |7879Exit codes: 0 ok, 1 lint failure or a case below `--threshold`, 2 argument80error, 3 `claude` CLI missing or too old, 5 stopped at the cap (partial81results kept). Results land in `<skill>/evals/results/`, which is gitignored.8283## The floor rules (why lint fails a suite)8485These come from the runner's own authoring guide. Each one exists because86without it the number stops meaning anything:8788- **At least one must-not-fire case.** A suite with only positive cases can't89 tell a skill that helps from one that fires on everything. The check is a90 `tool_used` grader with `tool: Skill`, `min: 0`, `max: 0`, **and**91 `arm: both` — leave out `arm` and it becomes display-only and never scores.92- **Every case has an outcome grader**, not just "the skill fired". A plain93 fired-check is excluded from the score under ablation; it can't move Δ.94 Grade what the user would notice: the answer, the file, the format.95- **`runs` ≥ 3** in the case file. Single runs are noise. For a pilot, pass96 `--runs 1` on the command line instead of lowering the floor in the file.97- **No absolute or `~/` paths** — cases run in a sandbox working directory.98- **Set `timeout_seconds` and `allowed_tools`** to fit the work. An99 under-set timeout scores 0, not "timed out", which reads as a broken skill.100 Tools beyond Read/Glob/Grep/Skill (Bash, Write, Edit, WebFetch, `mcp__*`)101 also need an operator grant at run time: `--allow-tools Write`.102103## Cases that can actually measure something104105A case can follow every floor rule and still be unable to show a difference.106The common ways that happens, all seen in real conversions:107108- **The input isn't in the sandbox.** evals.json prompts often point at109 `~/repos/x` or an "attached" doc. In the sandbox both arms just report "not110 found", and Δ ≈ 0 says nothing about the skill. Build a small fixture repo111 and copy it in with a `case.yaml` `scaffold_script`, then run with112 `--scaffold`. Put contradictions in the fixture (a doc claim the code113 disproves) so a grader can tell careful work from a skim.114- **The skill pauses to ask.** Eval sessions can't answer a question. If the115 skill confirms before acting (an output folder, a destructive step), state116 the answer in the prompt ("write to `./out/` — confirmed"). Otherwise the117 with-arm scores worse _for following its own rule_.118- **Writing skills get read-only defaults.** `convert` gives every case119 Read/Glob/Grep/Skill, 20 turns and 600 s, and its notes flag cases that look120 like they write files. Those need Write in `allowed_tools`, a real budget121 (deep-dive-sized work can take 60 turns and 1500 s), `--allow-tools Write`122 at run time, and graders that read the file (`regex` with123 `target: {source: file, path}`, or `file_exists`). A judge on the last124 message can't see what the skill wrote.125- **`--ablation none` isn't a cheaper Δ.** It drops the without-arm, so there126 is no Δ at all. To save money, pilot one case with `--runs 1`.127128## Grader choice is a cost choice129130| Type | Cost | Use for |131| -------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------ |132| `regex` | free | Mechanical claims: a ` ```mermaid ` block exists, a heading is present, a phrase is absent (`match: not_contains`) |133| `file_exists` | free | The skill must create a file (`path: <glob>`) |134| `tool_used` / `tool_order` | free | Fired / must-not-fire; tool sequencing |135| `llm` | paid judge | Claims a pattern can't check. Write the rubric as concrete checkable claims |136| `baseline` | paid judge | Preference between arms — rarely worth it |137138`convert` writes assertions into one `llm` rubric because evals.json139assertions are prose. After converting, rewrite any assertion that's really a140pattern as a free `regex` grader. The judge defaults to Haiku; the runner's141guide recommends `--judge-model sonnet` for nuanced rubrics, which costs more.142That trade is the owner's to make.143144The judge is the small cost. **Sessions are the big one**: a full run is145(cases × runs × 2 arms) complete Claude sessions, each doing real work. `lint`146prints that count so the size is visible before anything is spent.147148## The spend protocol149150Session runs follow the same rule as every paid call in this repo: estimate,151run the smallest version, never raise a cap without asking.1521531. `lint` — free. Read the session count.1542. **Pilot one case**: `run <skill> --case '<one>' --runs 1 --max-cost-usd 0.25`.155 Use the must-not-fire case first; it's the shortest.1563. `report` — real `costUsd` for those 2 sessions. Multiply by the session157 count from step 1 for the full suite, and tell the owner the figure158 (subscription usage, or dollars on an API key) before running it.1594. Full run only on the owner's yes, with a cap at that figure plus headroom.160 Exit 5 means the cap stopped it; report the partial result. Don't rerun161 with a higher cap on your own.162163**Trust.** The first run in a folder asks whether you trust the plugin; a164non-interactive run can't answer. Ask the owner once, and pass `--trust` only165after a yes. It doesn't grant extra tools or scaffold scripts.166167**Publishing.** `run` keeps the HTML report local (`--no-publish`). Pass168`--publish` only when the owner wants it on claude.ai.169170## Reading a result171172- **Δ > 0 on the skill's own cases** — it helps. Keep it.173- **Δ ≈ 0 where the skill claims value** — the model already does this174 unaided. That's a prune or rewrite candidate for `skill-update`, or a sign175 the cases are too easy (the model only reaches for skills on tasks it can't176 do trivially, so trivial prompts score the same with or without).177- **Must-not-fire case fails** — the description over-triggers. Take the178 prompt to `skill-triage collisions --skill <name>` to see who it's stealing179 from.180- **`report` warns a plugin `problem`** (`manifest_invalid`,181 `disabled_by_default`, `will_not_load`) — the with-arm ran without the skill.182 The scores are meaningless; fix loading first.183- **An implausible jump between runs** — suspect the judge being gamed;184 spot-check the transcripts before believing it.185186Re-run a skill's suite after a model or Claude Code upgrade. Harness187behaviour shifts, and `model-adaptation` pruning should be backed by a Δ, not188a hunch.189190## Anti-patterns191192| Anti-pattern | Why it fails |193| ---------------------------------------------------- | -------------------------------------------------------------------- |194| Running the full suite before a one-case pilot | You learn the cost after paying it |195| A suite with no must-not-fire case | Can't tell a useful skill from an over-eager one |196| `tool_used: Skill` as a case's only grader | Excluded from the score; Δ is always 0 |197| `llm` graders for mechanical checks | Paying a judge for what a regex does free |198| Lowering `runs` in the case file to save money | Breaks the floor; use `--runs 1` for pilots only |199| Raising `--max-cost-usd` after exit 5 without asking | The cap is the owner's decision |200| Absolute paths in prompts | The sandbox has a different cwd; the case fails for the wrong reason |