Validate eval cases before a run spends money
A case that fails to load costs the same as one that works. scripts/validate-cases.py reads the
suite off disk and reports what would break, with no model call and no network access.
Run it
python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>
<eval-dir> is the directory holding the cases, evals/ under the plugin root by default. Add
--json for the findings as an array of {level, case, file, message} objects; --help prints
the full contract from the script's header.
| Exit |
Meaning |
| 0 |
no FAIL finding. WARN findings are printed and do not gate |
| 1 |
at least one FAIL finding |
| 2 |
usage error, or the eval dir is missing or unreadable |
Findings print one per line as <FAIL|WARN> <case>/<file>: <message>, sorted by case then file; a
finding about the suite as a whole carries the eval dir in place of <case>/<file>.
What the two tiers mean
FAIL is what the binary itself rejects: an unknown prompt.md frontmatter key, a grader with no
usable type, an unknown option for the declared grader type, a case with no grader, two graders
sharing a name, a non-positive weight, runs / max_turns / timeout_seconds out of bounds, an
env key that does not match EVAL_[A-Z0-9_]*, a case that has a case.yaml but no
schema_version or name in it or in prompt.md frontmatter (without a case.yaml the binary
supplies both), a case with no prompt (prompt.md body or execution.prompt), a grader missing an
option its type requires, a grader or prompt.md whose frontmatter holds --- on a line other
than its two delimiter lines (the binary cannot load the file; spell it -{3} in a regex), a
plugins entry that resolves outside the plugin under evaluation or, inside it, to anything but
the plugin itself or a subdirectory of the case (see the containment record below), and a
schema_version that is not a quoted version string or whose major is newer than the binary
supports. The complete sets the script checks, so a question about a key or a type is answered
here without opening the script:
prompt.md frontmatter keys: schema_version, name, description, tags, plugins,
runs, expected_outcome, model, max_turns, timeout_seconds, allowed_tools,
append_system_prompt, env. A time limit is timeout_seconds; timeout is an unknown key, so
the case fails to load.
- Grader
type: regex, tool_order, tool_used, file_exists, llm, baseline. Any other
value, such as contains or string_match, is no usable type, so the case fails to load. A
substring or pattern check is type: regex with the text in pattern.
- Options per type, beside the
type, weight, arm and name every grader takes: regex
pattern, flags, match, target; tool_used tool, input_match, min, max;
tool_order before, after; file_exists path, exists; llm criteria, focus;
baseline baseline_file, criteria.
- Required options per type:
regex pattern; tool_used tool; tool_order before,
after; file_exists path; llm criteria; baseline baseline_file, criteria. A
graders/*.md body stands in for pattern (regex) or criteria (llm, baseline).
- Regex
target and llm focus values: last_message, trace, files, mock_calls, or
{ source: file, path: <path> }. Any other value, such as focus: reply, fails to load.
- Bounds:
runs 1 to 50, max_turns 1 to 200, timeout_seconds 1 to 3600.
The script's constants are the one place to correct when the schema moves, under the drift record
its header carries; this list follows them.
The history-file check is this skill's own FAIL, not a load rejection: a multi-record
context.history_file whose records do not form one uuid/parentUuid chain is a FAIL, and so
is a record that is not JSON. Chain the records: a uuid on each, parentUuid: null on the first
and the previous record's uuid on each later one. A record that is not a user, assistant or
system message and carries no uuid (a summary or file-history-snapshot line) is off the
chain and skipped. Why the chain matters is in the script's history-file drift record.
WARN is an authoring mistake the suite loads with and then scores badly on: target: files
(which reads the list of paths created, not their contents), an inline (?i) the grader's regex
engine does not honor, a case whose graders are all judges (the two paid types, with no
deterministic grader beside them), a tool in allowed_tools that the operator must grant with
--allow-tools, a case naming a skill (leading /<plugin>:<skill> or a tag) whose SKILL.md has a
pre-computed context line while allowed_tools lacks Bash, file_exists in a case that requests no write tool (so nothing it could match is
ever created), and a prompt.md key the binary loads but the reference page does not list, which
can change without notice (the script's UNDOCUMENTED_PROMPT_KEYS, under its own drift record).
The judge-only rule is this skill's own pairing heuristic, not a rejection the binary makes; the
sample-answer findings below are likewise this skill's checks.
A WARN never sets exit 1, so a suite can ship with warnings on purpose. Read them once and decide.
Sample answers
A grader that rejects a correct paraphrase scores the plugin down for nothing, and no run says the
grader was at fault. So each case can carry known answers, and the script runs the free graders
over them before any money is spent.
Put them in samples/<grader-name>.json inside the case directory, beside graders/ and never in
it:
{
"pass": [{"answer": "...", "why": "the ranking rule with 'and' for the comma"}],
"fail": [{"answer": ""}, {"answer": "...", "why": "near miss: base-model phrasing"}]
}
answer is what the grader reads: text for a regex (the final reply on the default target), a
list of {"tool", "input"} calls for tool_used and tool_order, and a list of created paths for
file_exists. why is optional and is echoed in any finding. Give each grader several paraphrases
it must pass, near misses it must reject, an empty answer, and an answer to a different question.
| Finding |
Level |
| A must-pass answer the grader rejects, or a must-fail answer it accepts |
FAIL |
| A sample file that is not JSON in this shape, or an answer of the wrong kind for its grader |
FAIL |
A regex, tool_used, tool_order or file_exists grader with no sample file |
WARN |
| A sample file with no must-pass or no must-fail answers, or named after no grader |
WARN |
A setting the script cannot reproduce offline, such as the y or v regex flag |
WARN |
A pattern with Python-only syntax: \A, \Z, \z, a possessive quantifier, an atomic group, a (?P< group or (?P= reference, a (?# comment, a (?( conditional. The finding names the grader and the token |
FAIL |
With node on PATH: a pattern or input_match that does not compile under JavaScript, or a sample JavaScript's RegExp sorts the other way |
FAIL |
An llm or baseline grader with samples: they need a paid judge calibration run |
WARN |
The script FAILs the Python-only tokens listed above; that check needs nothing installed. When
node is on PATH it also grades each regex grader and each tool_used input_match, with their
samples, through RegExp, and without node it skips that part silently. A grader built from the
shared matchers is checked by scripts/check-matchers.py, which requires node; the plugin
README's shared grader matchers section covers it.
The script never calls a judge. Samples on an llm grader are the labeled answers a calibration
run feeds the judge, and that run is the operator's to start.
- Pointer: the samples location rests on the case layout and run isolation in the
eval suite reference and on
the case loader in Claude Code 2.1.287, which reads only
prompt.md, case.yaml and graders/.
The sample check grades as the binary does, per its grader code read against the
grader types table; Python's re
stands in for the JavaScript regex engine, and the script's header lists where the two differ.
- As of: 2026-10-01
- Recheck trigger: the next Claude Code release, a release that adds a file the loader reads
from a case directory, or a grader whose sample verdict disagrees with its verdict in a real run.
When the parser stops
The script reads a bounded YAML subset: scalars, quoted scalars, flow lists and mappings, block
lists including lists of mappings, and block mappings three levels deep. An anchor, an alias, a tag,
a block scalar, a merge key, tab indentation, a nested sequence in either spelling, a backslash
escape outside \\, \", \/, \n, \t, \r, or nesting past three levels is reported as
frontmatter not parsed (<construct>) at FAIL, never waved through at exit 0, because a validator
that green-lights input it could not read is worse than no validator. Simplify the file, or run the
CLI, which carries the full parser.
Mirroring record
The load-time FAIL findings match what the binary rejects when it loads a case, so a suite at exit
0 loads.
- Pointer: the case schema recovered from the shipped Claude Code binary, read against the
eval-suite reference at https://code.claude.com/docs/en/plugin-evals#eval-suite-reference; the
required fields were re-derived from Claude Code 2.1.296 and each rejection reproduced with
claude plugin eval --case.
- As of: 2026-10-09
- Recheck trigger: the next Claude Code release, which can add a frontmatter key, add a grader
type, or move a bound. On a firing, re-derive both lists from the schema rather than patching
one value.
The plugins check takes the plugin under evaluation as the nearest directory at or above the
suite with a plugin manifest, else the suite's parent. A case that needs a second plugin loads a
stand-in from a subdirectory of its own directory; a stub shared between cases does not load.
This skill asserts nothing about whether claude plugin eval is available in your install; it
reads files and never invokes anything. /evals:plugin-eval preflight owns that question.
Boundary
This validator reads only the prompt.md / case.yaml / graders/*.md layout, the format
/evals:design writes for a skill. Pointed at a skill-creator JSON eval file, it reports only that
the directory holds no cases. Convert that file's cases to this layout (/evals:design skill <name>
writes them), then validate the case directories.
Next
/evals:plugin-eval <target>. A suite at exit 0 is one worth paying to run.
Gotchas
- Exit 0 means the suite loads, not that it measures anything. A case whose graders pass in both
arms proves the plugin contributed nothing; the delta is read after a run, not here.
- Samples that all sort correctly prove the grader handles those answers, not the ones the model
will write. When a real run fails a correct answer, add that answer's phrasing as a sample.
- A case is a directory holding
prompt.md or case.yaml. A directory holding neither is skipped
silently, so a misnamed prompt file reads as absent rather than as an error. An eval dir with no
case at all is a FAIL.
results/ and mocks/ are never treated as cases, whatever they contain.
- When both files exist,
prompt.md frontmatter overrides the matching case.yaml field, and the
bounds check reports the file the effective value came from, which may not be the file you edited.
- Unknown top-level keys in
case.yaml are not reported. The unknown-key rejection is recorded for
prompt.md, and this validator invents no rejection the binary does not make.
- The script is Python 3.8+, standard library only, so it runs from a consumer checkout where no
package is provisioned. It never writes to the suite it reads.
1---2name: validate3description: "Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample answers. Use when: 'validate my eval cases', 'check my eval suite', 'will this suite load', 'lint case.yaml', 'check my graders', 'why did my case fail to load', or before paying for a run. Not for the skill-creator JSON eval format."4argument-hint: "[eval-dir]"5user-invocable: true6disable-model-invocation: false7metadata:8 source: https://skillmd.com/skills/melodic-software/validate9 workflow-stage: test10 summary: Static FAIL/WARN check of an eval suite's cases and graders, with no model call11---1213# Validate eval cases before a run spends money1415A case that fails to load costs the same as one that works. `scripts/validate-cases.py` reads the16suite off disk and reports what would break, with no model call and no network access.1718## Run it1920```bash21python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>22```2324`<eval-dir>` is the directory holding the cases, `evals/` under the plugin root by default. Add25`--json` for the findings as an array of `{level, case, file, message}` objects; `--help` prints26the full contract from the script's header.2728| Exit | Meaning |29|---|---|30| 0 | no FAIL finding. WARN findings are printed and do not gate |31| 1 | at least one FAIL finding |32| 2 | usage error, or the eval dir is missing or unreadable |3334Findings print one per line as `<FAIL|WARN> <case>/<file>: <message>`, sorted by case then file; a35finding about the suite as a whole carries the eval dir in place of `<case>/<file>`.3637## What the two tiers mean3839**FAIL is what the binary itself rejects**: an unknown `prompt.md` frontmatter key, a grader with no40usable `type`, an unknown option for the declared grader type, a case with no grader, two graders41sharing a name, a non-positive `weight`, `runs` / `max_turns` / `timeout_seconds` out of bounds, an42`env` key that does not match `EVAL_[A-Z0-9_]*`, a case that has a `case.yaml` but no43`schema_version` or `name` in it or in `prompt.md` frontmatter (without a `case.yaml` the binary44supplies both), a case with no prompt (`prompt.md` body or `execution.prompt`), a grader missing an45option its type requires, a grader or `prompt.md` whose frontmatter holds `---` on a line other46than its two delimiter lines (the binary cannot load the file; spell it `-{3}` in a regex), a47`plugins` entry that resolves outside the plugin under evaluation or, inside it, to anything but48the plugin itself or a subdirectory of the case (see the containment record below), and a49`schema_version` that is not a quoted version string or whose major is newer than the binary50supports. The complete sets the script checks, so a question about a key or a type is answered51here without opening the script:5253- `prompt.md` frontmatter keys: `schema_version`, `name`, `description`, `tags`, `plugins`,54 `runs`, `expected_outcome`, `model`, `max_turns`, `timeout_seconds`, `allowed_tools`,55 `append_system_prompt`, `env`. A time limit is `timeout_seconds`; `timeout` is an unknown key, so56 the case fails to load.57- Grader `type`: `regex`, `tool_order`, `tool_used`, `file_exists`, `llm`, `baseline`. Any other58 value, such as `contains` or `string_match`, is no usable type, so the case fails to load. A59 substring or pattern check is `type: regex` with the text in `pattern`.60- Options per type, beside the `type`, `weight`, `arm` and `name` every grader takes: `regex`61 `pattern`, `flags`, `match`, `target`; `tool_used` `tool`, `input_match`, `min`, `max`;62 `tool_order` `before`, `after`; `file_exists` `path`, `exists`; `llm` `criteria`, `focus`;63 `baseline` `baseline_file`, `criteria`.64- Required options per type: `regex` `pattern`; `tool_used` `tool`; `tool_order` `before`,65 `after`; `file_exists` `path`; `llm` `criteria`; `baseline` `baseline_file`, `criteria`. A66 `graders/*.md` body stands in for `pattern` (regex) or `criteria` (llm, baseline).67- Regex `target` and llm `focus` values: `last_message`, `trace`, `files`, `mock_calls`, or68 `{ source: file, path: <path> }`. Any other value, such as `focus: reply`, fails to load.69- Bounds: `runs` 1 to 50, `max_turns` 1 to 200, `timeout_seconds` 1 to 3600.7071The script's constants are the one place to correct when the schema moves, under the drift record72its header carries; this list follows them.7374The history-file check is this skill's own FAIL, not a load rejection: a multi-record75`context.history_file` whose records do not form one `uuid`/`parentUuid` chain is a FAIL, and so76is a record that is not JSON. Chain the records: a `uuid` on each, `parentUuid: null` on the first77and the previous record's `uuid` on each later one. A record that is not a `user`, `assistant` or78`system` message and carries no `uuid` (a `summary` or `file-history-snapshot` line) is off the79chain and skipped. Why the chain matters is in the script's history-file drift record.8081**WARN is an authoring mistake the suite loads with** and then scores badly on: `target: files`82(which reads the list of paths created, not their contents), an inline `(?i)` the grader's regex83engine does not honor, a case whose graders are all judges (the two paid types, with no84deterministic grader beside them), a tool in `allowed_tools` that the operator must grant with85`--allow-tools`, a case naming a skill (leading `/<plugin>:<skill>` or a tag) whose SKILL.md has a86pre-computed context line while `allowed_tools` lacks Bash, `file_exists` in a case that requests no write tool (so nothing it could match is87ever created), and a `prompt.md` key the binary loads but the reference page does not list, which88can change without notice (the script's `UNDOCUMENTED_PROMPT_KEYS`, under its own drift record).89The judge-only rule is this skill's own pairing heuristic, not a rejection the binary makes; the90sample-answer findings below are likewise this skill's checks.9192A WARN never sets exit 1, so a suite can ship with warnings on purpose. Read them once and decide.9394## Sample answers9596A grader that rejects a correct paraphrase scores the plugin down for nothing, and no run says the97grader was at fault. So each case can carry known answers, and the script runs the free graders98over them before any money is spent.99100Put them in `samples/<grader-name>.json` inside the case directory, beside `graders/` and never in101it:102103```json104{105 "pass": [{"answer": "...", "why": "the ranking rule with 'and' for the comma"}],106 "fail": [{"answer": ""}, {"answer": "...", "why": "near miss: base-model phrasing"}]107}108```109110`answer` is what the grader reads: text for a `regex` (the final reply on the default target), a111list of `{"tool", "input"}` calls for `tool_used` and `tool_order`, and a list of created paths for112`file_exists`. `why` is optional and is echoed in any finding. Give each grader several paraphrases113it must pass, near misses it must reject, an empty answer, and an answer to a different question.114115| Finding | Level |116|---|---|117| A must-pass answer the grader rejects, or a must-fail answer it accepts | FAIL |118| A sample file that is not JSON in this shape, or an answer of the wrong kind for its grader | FAIL |119| A `regex`, `tool_used`, `tool_order` or `file_exists` grader with no sample file | WARN |120| A sample file with no must-pass or no must-fail answers, or named after no grader | WARN |121| A setting the script cannot reproduce offline, such as the `y` or `v` regex flag | WARN |122| A pattern with Python-only syntax: `\A`, `\Z`, `\z`, a possessive quantifier, an atomic group, a `(?P<` group or `(?P=` reference, a `(?#` comment, a `(?(` conditional. The finding names the grader and the token | FAIL |123| With `node` on PATH: a `pattern` or `input_match` that does not compile under JavaScript, or a sample JavaScript's `RegExp` sorts the other way | FAIL |124| An `llm` or `baseline` grader with samples: they need a paid judge calibration run | WARN |125126The script FAILs the Python-only tokens listed above; that check needs nothing installed. When127`node` is on PATH it also grades each regex grader and each `tool_used` `input_match`, with their128samples, through `RegExp`, and without `node` it skips that part silently. A grader built from the129shared matchers is checked by `scripts/check-matchers.py`, which requires `node`; the plugin130README's shared grader matchers section covers it.131132The script never calls a judge. Samples on an `llm` grader are the labeled answers a calibration133run feeds the judge, and that run is the operator's to start.134135- **Pointer**: the samples location rests on the case layout and run isolation in the136 [eval suite reference](https://code.claude.com/docs/en/plugin-evals#eval-suite-reference) and on137 the case loader in Claude Code 2.1.287, which reads only `prompt.md`, `case.yaml` and `graders/`.138 The sample check grades as the binary does, per its grader code read against the139 [grader types](https://code.claude.com/docs/en/plugin-evals#grader-types) table; Python's `re`140 stands in for the JavaScript regex engine, and the script's header lists where the two differ.141- **As of**: 2026-10-01142- **Recheck trigger**: the next Claude Code release, a release that adds a file the loader reads143 from a case directory, or a grader whose sample verdict disagrees with its verdict in a real run.144145## When the parser stops146147The script reads a bounded YAML subset: scalars, quoted scalars, flow lists and mappings, block148lists including lists of mappings, and block mappings three levels deep. An anchor, an alias, a tag,149a block scalar, a merge key, tab indentation, a nested sequence in either spelling, a backslash150escape outside `\\`, `\"`, `\/`, `\n`, `\t`, `\r`, or nesting past three levels is reported as151`frontmatter not parsed (<construct>)` at FAIL, never waved through at exit 0, because a validator152that green-lights input it could not read is worse than no validator. Simplify the file, or run the153CLI, which carries the full parser.154155## Mirroring record156157The load-time FAIL findings match what the binary rejects when it loads a case, so a suite at exit1580 loads.159160- **Pointer**: the case schema recovered from the shipped Claude Code binary, read against the161 eval-suite reference at <https://code.claude.com/docs/en/plugin-evals#eval-suite-reference>; the162 required fields were re-derived from Claude Code 2.1.296 and each rejection reproduced with163 `claude plugin eval --case`.164- **As of**: 2026-10-09165- **Recheck trigger**: the next Claude Code release, which can add a frontmatter key, add a grader166 type, or move a bound. On a firing, re-derive both lists from the schema rather than patching167 one value.168169The `plugins` check takes the plugin under evaluation as the nearest directory at or above the170suite with a plugin manifest, else the suite's parent. A case that needs a second plugin loads a171stand-in from a subdirectory of its own directory; a stub shared between cases does not load.172173- **Pointer**: the CLI's load refusal for a `plugins` entry outside its containment root, in the174 "Run the plugin eval" step log of175 <https://github.com/melodic-software/claude-code-plugins/actions/runs/41894661524>; its refusal176 of a stub shared between cases, in the Fix section of177 <https://github.com/melodic-software/claude-code-plugins/pull/13376>.178- **As of**: 2026-11-04179- **Recheck trigger**: a Claude Code release that changes that refusal message or how the180 containment root is chosen.181182This skill asserts nothing about whether `claude plugin eval` is available in your install; it183reads files and never invokes anything. `/evals:plugin-eval` preflight owns that question.184185## Boundary186187This validator reads only the `prompt.md` / `case.yaml` / `graders/*.md` layout, the format188`/evals:design` writes for a skill. Pointed at a skill-creator JSON eval file, it reports only that189the directory holds no cases. Convert that file's cases to this layout (`/evals:design skill <name>`190writes them), then validate the case directories.191192## Next193194`/evals:plugin-eval <target>`. A suite at exit 0 is one worth paying to run.195196## Gotchas197198- Exit 0 means the suite **loads**, not that it measures anything. A case whose graders pass in both199 arms proves the plugin contributed nothing; the delta is read after a run, not here.200- Samples that all sort correctly prove the grader handles those answers, not the ones the model201 will write. When a real run fails a correct answer, add that answer's phrasing as a sample.202- A case is a directory holding `prompt.md` or `case.yaml`. A directory holding neither is skipped203 silently, so a misnamed prompt file reads as absent rather than as an error. An eval dir with no204 case at all is a FAIL.205- `results/` and `mocks/` are never treated as cases, whatever they contain.206- When both files exist, `prompt.md` frontmatter overrides the matching `case.yaml` field, and the207 bounds check reports the file the effective value came from, which may not be the file you edited.208- Unknown top-level keys in `case.yaml` are not reported. The unknown-key rejection is recorded for209 `prompt.md`, and this validator invents no rejection the binary does not make.210- The script is Python 3.8+, standard library only, so it runs from a consumer checkout where no211 package is provisioned. It never writes to the suite it reads.