Design

Design an evaluation suite

Guides the consumer from "I want to evaluate X" to committed artifacts: a success-criteria document and a graded eval suite. Method follows Anthropic's official evaluation guidance — load /evals:methodology reference files as each phase needs them (they carry the distilled source).

Arguments

$ARGUMENTS names the target. Two shapes:

• app (or a path/description of an LLM-powered feature) — evals for the consumer's own LLM-based application behavior.
• skill <name> — evals for a consumer-authored Claude Code skill, emitted as evals/evals.json next to that skill.

No argument → ask which target, with one example of each.

Phase 1 — success criteria (before any cases)

Interview until each criterion is specific, measurable, achievable, relevant (success-criteria.md):

1. What does success look like, concretely? Reject unmeasurable phrasings by proposing a measurable rewrite ("good answers" → "≥90% of answers judged correct against their rubric").
2. Which dimensions matter? Walk the eight (fidelity, consistency, relevance/coherence, tone/style, privacy, context use, latency, price); keep the ones with a real user need. Most targets are multidimensional — press for at least fidelity plus one guardrail dimension.
3. What is achievable? Anchor each target to a baseline (current behavior, prior experiment, or a published benchmark); when no baseline exists, record the first run AS the baseline.

Write the result to docs/eval-criteria/<target>.md in the consumer repo (create the directory if absent; respect an existing consumer convention for criteria docs if one is documented in the consumer's own CLAUDE.md or rules). Each criterion: dimension, metric, target number/scale, rationale line.

Phase 2 — eval suite

Per criterion, pick the cheapest reliable grading method (grading.md, recipes.md): code-graded where the output can be constrained to allow it; LLM-graded with a tight rubric and constrained verdict otherwise; human grading only with stated justification.

Case authoring (eval-design.md):

• Mirror the target's real input distribution; include edge cases explicitly — irrelevant or nonexistent input, overly long input, poor/harmful/irrelevant user input for chat surfaces, ambiguous cases.
• Every case carries a golden answer: an exact answer for code-graded cases, rubric-instructions for LLM/human-graded cases.
• Draft a baseline set by hand with the consumer, then offer to generate more cases from it — volume over polish — and have the consumer review the generated batch before it lands.

Target = app: scaffold evals/<target>/cases.jsonl (one JSON object per case: id, input, golden_answer, grading (exact|string_match|llm_rubric|human), optional rubric) plus a README.md documenting how the consumer's own tooling should run and grade them, with the grader prompt skeleton from grading.md inlined for llm_rubric cases. Honor an existing consumer eval layout when one is already present — extend, don't rename.

Target = skill: emit <skills-root>/<skill>/evals/evals.json in this shape — skill_name, evals[] of {id, name (kebab-case), prompt, expected_output, expectations[]} — covering trigger/routing, the happy path, at least one refusal/guardrail, and one anti-pattern the skill must not exhibit. When the skill-quality plugin is installed, validate with /skill-quality:check validate-evals <skill> (its bundled schema is the contract); otherwise state that the file follows the marketplace's evals schema and validation was skipped.

Phase 3 — grading hygiene gate

Before finishing, confirm and record in the criteria doc:

• LLM-graded cases name a grader model DIFFERENT from the generator and constrain the verdict format. Add a reasoning-then-discard instruction only where the grader model does not already think before answering; a grader with always-on thinking reasons before it decides, so an output-side reasoning block buys nothing and roughly doubles the output tokens every re-run pays for.
• The consumer's first act is to sample-check grader verdicts against their own judgment before trusting the suite at scale.
• Re-run cost is stated (which cases are code-graded and free vs LLM-graded and metered).

What this skill does NOT do

• Does not execute evals. No marketplace command runs model-graded evals; running the suite is the consumer's tooling (for Claude Code skills, Anthropic's skill-creator plugin can run skill evals when installed).
• Does not overwrite an existing criteria doc or eval suite without showing the diff and getting explicit confirmation.
• Does not invent baselines — a target with no anchor is recorded as provisional.

Next

/skill-quality:check validate-evals <skill>. An emitted evals.json is what it validates.

Gotchas

• A consumer saying "just write some tests" still gets Phase 1 — criteria first is the method, not a preference; keep it to the few questions that unblock measurable targets.
• Refuse to emit an eval case with no golden answer or rubric — a case that can't be graded is not an eval.