Skill Evaluating

Evaluating Skills

Eval methodology for skills, based on agentskills.io and Anthropic's eval guide.

Gotchas

• Assertions written before observing output are almost always wrong: run the eval first, observe actual output, THEN write assertions
• Near-miss negative test cases are the most valuable: prompts that share keywords with should-trigger cases but need a different skill entirely
• Select the best skill iteration by validation pass rate, not the last one
• Grade outcomes, not execution paths: if the agent solved the task via an unexpected route but produced correct output, that is a pass
• Same skill can behave differently across agent frameworks: test on each target agent (Claude Code, Hermes, OpenClaw, Codex)

Procedure

Step 1: Choose eval scope

Determine what you are evaluating:

• Description triggering: does the agent activate the right skill for a given prompt? Tests the description field effectiveness.
• Output quality: does the skill produce correct results when activated? Tests the skill body and references.
• Convention compliance: does the output follow the skill-creating standard? Read references/convention-checks.md for the full assertion library.

Multiple scopes can apply to the same skill. Start with the scope that addresses your immediate concern, add others as needed.

Step 2: Design test cases

Create evals/evals.json in the skill being evaluated. Copy the starter template from assets/eval-template.json in this skill.

For description triggering:

• Write ~20 queries: 8-10 should-trigger, 8-10 should-not-trigger
• Split 60/40 into train and validation sets (proportional mix in each)
• Include near-miss negatives that share keywords but need a different skill
• In expected_output, describe whether the skill should or should not activate and why

For output quality:

• Start with 2-3 test cases, expand after first results
• Use realistic user prompts with varied phrasing, detail level, and formality
• In expected_output, describe what correct output looks like, not exact text
• Use files array for any input files the test needs

For convention compliance:

• Start with 2-3 test cases targeting specific convention areas
• In expected_output, describe which conventions must be met
• Read references/convention-checks.md for the full assertion library

Leave assertions arrays empty. They are populated after Step 4.

Step 3: Run paired evals

Dispatch a subagent per test case to ensure clean context: no leakage between test runs. Run each case twice:

1. With skill: the skill is available and loaded
2. Without skill: the skill is not available (baseline comparison)

The delta between with-skill and without-skill measures what the skill adds. If the delta is negligible, the skill may not be adding value for that case.

Grade outcomes, not paths. An unexpected tool-call sequence that produces correct output is still a pass.

Step 4: Write assertions after observing

Review actual outputs from Step 3. Write specific, verifiable assertions based on what you observed, not what you expected beforehand.

Choose the grader type that fits each assertion:

• Code-based: structure, file existence, format validity, counts
• LLM-as-judge: quality, completeness, tone, semantic correctness
• Human: edge cases, calibration, judgment calls

Read references/eval-methodology.md for the full grader taxonomy and LLM-judge calibration protocol.

Good assertions are specific, verifiable, and countable: "Output includes at least 3 concrete recommendations."

Weak assertions are vague: "Output is good." Brittle assertions use exact phrase matching.

Require concrete evidence for PASS. No benefit of the doubt.

Step 5: Grade and iterate

Compute pass rates per test case and aggregate across the eval suite.

Track two metrics separately:

• pass@k: succeeded at least once in k trials (capability)
• pass^k: succeeded every time in k trials (reliability)

Read references/eval-methodology.md for formulas and interpretation.

Three signal sources for skill improvement:

1. Failed assertions: specific gaps in the skill
2. Human feedback: broader quality issues not captured by assertions
3. Execution transcripts: reveals WHY things went wrong

Feed all three plus the current SKILL.md to propose targeted changes. Iterate on the train set only. Check the validation set for generalization. 5 iterations is usually enough. If not improving after 5, the test cases themselves may be the problem: revisit Step 2.

Step 6: Graduate and monitor

Once a skill hits ~100% on capability evals (pass@k = 1.0 for 3+ consecutive runs):

1. Graduate capability evals into regression tests: run them on every skill change to protect against backsliding
2. Monitor the with/without delta over time: if it shrinks to zero across 3+ runs, the model may have internalized the skill
3. Retire cautiously: archive the skill, do not delete. Re-test after model updates in case capabilities regress.

Read references/eval-methodology.md for graduation criteria, retirement detection, and the three-layer eval taxonomy.