# Design > Design an evaluation suite for an LLM-based application or a Claude Code skill: interview for measurable success criteria, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: 'design evals', 'create an eval suite', 'scaffold evals', 'write evals for my skill', 'define success criteria for this app', 'set up LLM testing', 'build a test set for my prompt' — not for eval-design theory questions (use /evals:methodology), not for statically validating an existing evals.json (use /skill-quality:check validate-evals when installed), and it does not execute evals. - Skill: `melodic-software/design` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add melodic-software/design` - Raw SKILL.md: https://api.skillmd.com/api/skills/melodic-software/design/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: melodic-software (https://skillmd.com/u/melodic-software) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/melodic-software/design --- # Design an evaluation suite Guides the consumer from "I want to evaluate X" to committed artifacts: a success-criteria document and a graded eval suite. Method follows Anthropic's official evaluation guidance — load `/evals:methodology` reference files as each phase needs them (they carry the distilled source). ## Arguments `$ARGUMENTS` names the target. Two shapes: - **`app`** (or a path/description of an LLM-powered feature) — evals for the consumer's own LLM-based application behavior. - **`skill `** — evals for a consumer-authored Claude Code skill, emitted as `evals/evals.json` next to that skill. No argument → ask which target, with one example of each. ## Phase 1 — success criteria (before any cases) Interview until each criterion is **specific, measurable, achievable, relevant** ([success-criteria.md](../methodology/reference/success-criteria.md)): 1. What does success look like, concretely? Reject unmeasurable phrasings by proposing a measurable rewrite ("good answers" → "≥90% of answers judged correct against their rubric"). 2. Which dimensions matter? Walk the eight (fidelity, consistency, relevance/coherence, tone/style, privacy, context use, latency, price); keep the ones with a real user need. Most targets are multidimensional — press for at least fidelity plus one guardrail dimension. 3. What is achievable? Anchor each target to a baseline (current behavior, prior experiment, or a published benchmark); when no baseline exists, record the first run AS the baseline. Write the result to `docs/eval-criteria/.md` in the consumer repo (create the directory if absent; respect an existing consumer convention for criteria docs if one is documented in the consumer's own `CLAUDE.md` or rules). Each criterion: dimension, metric, target number/scale, rationale line. ## Phase 2 — eval suite Per criterion, pick the cheapest reliable grading method ([grading.md](../methodology/reference/grading.md), [recipes.md](../methodology/reference/recipes.md)): code-graded where the output can be constrained to allow it; LLM-graded with a tight rubric and constrained verdict otherwise; human grading only with stated justification. Case authoring ([eval-design.md](../methodology/reference/eval-design.md)): - Mirror the target's real input distribution; include edge cases explicitly — irrelevant or nonexistent input, overly long input, poor/harmful/irrelevant user input for chat surfaces, ambiguous cases. - Every case carries a golden answer: an exact answer for code-graded cases, rubric-instructions for LLM/human-graded cases. - Draft a baseline set by hand with the consumer, then offer to generate more cases from it — volume over polish — and have the consumer review the generated batch before it lands. **Target = app:** scaffold `evals//cases.jsonl` (one JSON object per case: `id`, `input`, `golden_answer`, `grading` (`exact|string_match|llm_rubric|human`), optional `rubric`) plus a `README.md` documenting how the consumer's own tooling should run and grade them, with the grader prompt skeleton from [grading.md](../methodology/reference/grading.md) inlined for `llm_rubric` cases. Honor an existing consumer eval layout when one is already present — extend, don't rename. **Target = skill:** emit `//evals/evals.json` in this shape — `skill_name`, `evals[]` of `{id, name (kebab-case), prompt, expected_output, expectations[]}` — covering trigger/routing, the happy path, at least one refusal/guardrail, and one anti-pattern the skill must not exhibit. When the `skill-quality` plugin is installed, validate with `/skill-quality:check validate-evals ` (its bundled schema is the contract); otherwise state that the file follows the marketplace's evals schema and validation was skipped. ## Phase 3 — grading hygiene gate Before finishing, confirm and record in the criteria doc: - LLM-graded cases name a grader model DIFFERENT from the generator and constrain the verdict format. Add a reasoning-then-discard instruction only where the grader model does not already think before answering; a grader with always-on thinking reasons before it decides, so an output-side reasoning block buys nothing and roughly doubles the output tokens every re-run pays for. - The consumer's first act is to sample-check grader verdicts against their own judgment before trusting the suite at scale. - Re-run cost is stated (which cases are code-graded and free vs LLM-graded and metered). ## What this skill does NOT do - **Does not execute evals.** No marketplace command runs model-graded evals; running the suite is the consumer's tooling (for Claude Code skills, Anthropic's `skill-creator` plugin can run skill evals when installed). - Does not overwrite an existing criteria doc or eval suite without showing the diff and getting explicit confirmation. - Does not invent baselines — a target with no anchor is recorded as provisional. ## Next `/skill-quality:check validate-evals `. An emitted `evals.json` is what it validates. ## Gotchas - A consumer saying "just write some tests" still gets Phase 1 — criteria first is the method, not a preference; keep it to the few questions that unblock measurable targets. - Refuse to emit an eval case with no golden answer or rubric — a case that can't be graded is not an eval.