Skill Evaluation

Skill Evaluation

If you need the vocabulary and tests behind Axes 1, 3, and 4 (leading words, completion criteria, context pointers, the deletion test, failure-mode definitions), read references/mechanics.md before scoring those axes.

Source

• Lessons from building Claude Code: How we use skills — Anthropic, Jun 2026
• "The Missing Manual: How to Write Great Skills" — Matt Pocock, AI Engineer World's Fair 2026 (video), and his writing-great-skills skill

Parameters

Also runs unattended: in CI, point target at skills changed in a PR and gate with scripts/score.py --fail-below 60 ... — non-zero exit below the threshold fails the check.

Criteria

18 criteria: 14 core, scored on every skill, plus 4 conditional criteria scored only when the skill's category makes them apply — otherwise mark N/A and exclude the criterion from both the numerator and denominator of the weighted average. Every score is 0–100 with evidence citing file, section, or line.

Axis 1 — Trigger (invocation)

Axis 2 — Structure

Axis 3 — Steering

Axis 4 — Pruning

Conditional criteria

Score only when the skill's category (from references/categories.md) makes the criterion apply; otherwise mark N/A and drop it from the weighted average entirely.

Override this table with judgment, in either direction: score a criterion for a skill outside these categories when it would clearly benefit (e.g., a non-product-verification skill that obviously needs a helper script), and mark it N/A even within an applicable category when the pattern doesn't fit the skill's shape (e.g., a pure-reference vocabulary skill filed under code-quality-and-review has nothing for a hook to enforce). Explain the override in the scorecard either way.

Overall score

overall = sum(score × weight) / sum(weight)

N/A criteria are excluded from both sums — never scored as 0, never counted as weight.

Scoring Guide

Grade Scale

Workflow

1. Read the target skill — SKILL.md, its frontmatter (check for disable-model-invocation), and every file in the skill directory.

2. Read references/mechanics.md — the vocabulary and tests Axes 1, 3, and 4 depend on, including what makes a context pointer's wording effective.

3. Classify — use references/categories.md and its decision tree to assign a category. The category determines which conditional criteria apply.

4. Score all applicable criteria — cite-or-cut: a criterion is only scored once its justification cites specific evidence (file, section, or line); no citation, no score. Mark N/A wherever the conditional table, or your own judgment, says a criterion doesn't apply. Done when every applicable criterion carries a score and a citation, and every N/A a reason.

5. Trigger eval — empirical test of whether the skill's description actually causes invocation. See the Trigger Eval section below for the full mechanic. Skip this step for user-invoked skills (disable-model-invocation: true) — they have no description to test.

6. Diagnose failure modes — done when every mode in the table below has been checked against the skill and either cited (file:line) or dismissed.

7. Assess bonus patterns — the 4 carried over from v1, plus a fifth:

   Bonus	Applies when	What to look for
   Validation loops	Skill produces output or modifies state	Instructs the agent to self-check before finalizing
   Output templates	Skill generates structured output	Includes a concrete template/example of expected format
   Procedures over declarations	Skill teaches a method	Teaches how to approach problems, not what to produce for one case
   Defaults over menus	Skill offers tool/approach choices	Picks a clear default, mentions alternatives briefly
   Trace-checkable steering	Skill uses leading words	The leading words are distinctive enough that a user could grep the agent's reasoning traces to confirm the skill actually fired

   Report each as Present / Absent / N/A.

8. Compute the weighted score — run scripts/score.py with one criterion:score:weight triple per criterion (score NA to exclude); it prints both sums, the overall, and the grade. Don't do this arithmetic by hand.

9. Write the scorecard to the output path — read references/output-template.md first (it also holds the comparison-mode template used when compare is set) and emit exactly that structure.

Trigger Eval

Empirical test of whether the skill's description causes a model to invoke it when it should — and ignore it when it shouldn't. This is not a pass/fail gate; it produces observational data that feeds the scorecard and informs the failure-mode diagnosis.

When to run

• Model-invoked skills only. User-invoked skills (disable-model-invocation: true) have no description to test — skip and mark the section N/A.

Prompt generation

Generate 10 prompts from the skill's description, scope, and gotchas:

• 5 should-trigger — realistic user requests that fall squarely within the skill's stated scope. Vary phrasing: some use the skill's vocabulary, others describe the same need in naive/indirect language.
• 5 should-not-trigger — requests that are adjacent but clearly outside scope (e.g., a sibling skill's territory, a task the description explicitly excludes, or a generic request a model handles without any skill).

Each prompt should read like something a real user would type — no meta-language about skills, no hints.

Sub-agent execution

Run each prompt in an independent sub-agent session with the target skill available. The sub-agent receives a single additional instruction appended to its system context:

At the end of your response, output exactly one line in this format:
SKILLS_USED: <comma-separated list of skill names you loaded during this task, or "none">

This instruction is generic — it does not name the skill under test or hint at what should be triggered. The sub-agent operates normally; it either loads the skill or doesn't based on the prompt alone.

Detection

Parse the SKILLS_USED: line from each sub-agent's response. Record per prompt:

What to report

Report raw counts — no pass/fail judgment:

• Should-trigger hit rate — X/5 triggered
• Should-not-trigger leak rate — X/5 triggered (lower is better)
• Other skills observed — which siblings fired on the same prompts

These numbers feed criterion #1 (invocation design) and #2 (description quality) with empirical evidence, and may reveal failure modes like over-triggering or description weakness.

Practical notes

• If the evaluation environment cannot spawn sub-agents (e.g., CI without agent access), skip the trigger eval and note "trigger eval: skipped (no agent access)" in the scorecard.
• A single trial per prompt is acceptable given the observational (non-gating) nature. Run multiple trials only if results are ambiguous.
• Keep prompts in the scorecard output so the skill author can reuse them as a regression set.

Failure-mode diagnosis

Name the failure mode, cite evidence, prescribe the defense. Each mode's defense is defined once in references/mechanics.md §5 — prescribe from there. This replaces a generic "top improvements" list.

After the table, write a Prioritized Actions section: 3–5 highest-impact actions derived directly from the detected failure modes, each citing its evidence.

Note: context overload — too many model-invoked skills competing for attention in one environment — is a portfolio-level problem, out of scope for evaluating a single skill. Record the description's context-load cost when it's notable; don't score the portfolio.

Gotchas

• Tiny skills (under ~50 lines) flood the scorecard with N/A — score what's there; a small, sharp skill can reach grade A on few criteria.
• Self-evaluation bias: when the skill under review is one you (or this session) wrote, apply the deletion test with extra skepticism — you will want your own lines to matter.
• Fresh rewrites still carry duplication: sediment needs time to settle, but duplication can ship on day one. Run the pruning axis even on brand-new skills.

Quality Checklist

Final gate before delivering — each item names the step whose completion it re-checks, nothing new:

• cite-or-cut held everywhere (step 4)
• every N/A justified (step 4)
• trigger eval run or skipped with reason (step 5)
• every failure mode cited or dismissed (step 6)
• 5 bonus patterns assessed (step 7)
• score computed by scripts/score.py, not by hand (step 8)