AI Skill Eval

ai-skill-eval

The ai-skill-eval CLI ships on $PATH via this dotfiles repo. It exists so that any SKILL.md edit can be empirically re-tested against a real model - most usefully a cheap local one (default: llama-cpp/qwen3-6-35b-a3b via pi -p) so the author gets fast signal on whether a small model can actually follow what the skill says.

When to reach for this skill

• You drafted a new SKILL.md and want proof a small model reads it correctly.
• You edited an existing skill's WHAT / WHEN / DO-NOT and want to confirm the change didn't regress trigger detection.
• Someone asks "does this skill still work against model X?" - run the eval with --model X.
• You want to extend coverage by adding more scenarios to an existing skill's evals/evals.json.

Skip this skill when:

• You're running a one-off smoke test during drafting and don't care about a persisted grade.
• The skill under question has no verifiable behavior (pure style guide with no triggerable action).
• You don't have time for a model round-trip and a manual read of the skill is enough signal.

Prerequisites

1. ai-skill-eval is on $PATH (installed by this dotfiles repo; verify with command -v ai-skill-eval).
2. A driver is available - either pi or claude on $PATH, or a --driver-cmd string you supply.
3. The skill under test has a sibling evals/evals.json (see schema below). If not, author it first.
4. node ≥ 24 is on $PATH - the CLI is a TypeScript executable that relies on Node's built-in type-stripping (present in this repo's test Docker image).

Authoring evals (the heart of skill validation)

Each skill gets one evals/evals.json file sibling to its SKILL.md. The file has two kinds of entries per skill:

• Positive eval: a realistic scenario where the skill's WHEN clause should fire.
• Negative eval: a near-miss scenario where the skill should not fire (prevents over-triggering).

{
  "skill_name": "plugin-conventions",
  "evals": [
    {
      "id": "positive-1",
      "should_trigger": true,
      "prompt": "I want to add a new plugin for the 'zoxide' CLI. It should set _ZO_DATA_DIR if unset and add a 'cd' wrapper. What conventions do I need to follow?",
      "expectations": [
        "The response names the 'command -v zoxide &>/dev/null || return' guard as mandatory.",
        "The response suggests a 10- or 30- numeric prefix (not a brand-new tier).",
        "The response mentions DOT_PLUGIN_DISABLE_zoxide as the disable switch."
      ]
    },
    {
      "id": "negative-1",
      "should_trigger": false,
      "prompt": "The user wants to add 'export LANG=en_US.UTF-8' to the shared dotfiles environment. No new tool.",
      "expectations": [
        "The response recognizes this is a phase-file edit, not a plugin.",
        "The response does not propose creating a new plugin file."
      ]
    }
  ]
}

Rules of thumb for writing evals:

• Prompts read like real user messages. "Please fix the bug on line 42" beats "Scenario: bug at line X".
• Expectations name concrete artifacts. Backtick-quote file paths, command names, and flags - the deterministic grader keyword-matches on those. Natural-language expectations need --critic-cmd to judge well.
• Negative evals are load-bearing. A skill that fires on everything is as broken as one that never fires. Ship a negative case for every positive.
• Two evals is a smoke test; 8–10 is a benchmark. Scale coverage to the stakes of the skill.

Core commands

Discover what's available

ai-skill-eval list

Prints the discovered skills and how many evals each has. Scan roots default to .agents/skills, config/agents/skills, config/pi/skills, and .claude/skills (any that exist in cwd). Override with --skill-root DIR (repeatable).

Run and grade everything

ai-skill-eval run

For each discovered skill that has evals/evals.json:

1. Builds the prompt: skill body + scenario + instruction to emit TRIGGER: / REASON: / NEXT_STEP:.
2. Invokes the driver (default: pi -p ... --model llama-cpp/qwen3-6-35b-a3b --no-session).
3. Parses the reply, grades TRIGGER exact-match and expectations via keyword-match.
4. Writes a markdown report to stdout.

Expected first run: ~15–20 seconds per eval against qwen3 on the local llama-cpp server.

Narrow the run

After a skill revision, run <skill> is the normal path (full skill re-validation). Reach for rerun only when you want to target a single failed eval.

# After editing a skill: re-validate the whole skill (both positive + negative evals).
ai-skill-eval run plugin-conventions

# One eval across all skills (filter by eval id):
ai-skill-eval run --only positive-1

# One specific failed eval after a skill revision (the fast-iteration loop):
ai-skill-eval rerun plugin-conventions:positive-1

# Use a different model:
ai-skill-eval run --model llama-cpp/some-other-local

# Use claude instead of pi:
ai-skill-eval run --driver claude --model claude-haiku-4-5

# Use codex (OpenAI CLI):
ai-skill-eval run --driver codex --model gpt-5-codex

# Custom driver (any command that reads $AI_SKILL_EVAL_PROMPT_FILE and prints the reply):
ai-skill-eval run --driver-cmd 'ollama run llama3 < "$AI_SKILL_EVAL_PROMPT_FILE"'

rerun REQUIRES the SKILL:EVAL_ID form (colon + eval id) - rerun plugin-conventions alone is a usage error. If you want to re-run the whole skill, use run plugin-conventions instead.

Re-grade without re-running

After tweaking expectations or a critic prompt:

ai-skill-eval grade plugin-conventions

Uses the stored results under .ai-skill-eval/plugin-conventions/results/*.txt; does not call the driver.

Subjective grading with a critic

The default grader is a lower-bound keyword-match. To get real judgment on expectation prose, plug in a critic:

ai-skill-eval run --critic-cmd 'claude -p "$(cat "$AI_SKILL_EVAL_PROMPT_FILE")" --model claude-haiku-4-5 --bare'

The critic is sent a prompt containing the skill, eval, model reply, and expectations, and must return JSON:

{
  "expectations": [{ "text": "...", "passed": true, "evidence": "..." }],
  "flaws": ["..."]
}

Any critic driver is fine as long as it writes JSON on stdout. The critic's verdict overrides the keyword-match verdict in each eval's grade file.

Report existing results

ai-skill-eval report                  # markdown
ai-skill-eval report --json           # machine-readable
ai-skill-eval report plugin-conventions

Interpreting the output

Every grade file includes:

• trigger_pass - hard signal. If this is ever false, the skill's WHEN clause is ambiguous. Revise before shipping.
• expectation_pass / expectation_total - soft signal under deterministic grading; hard signal under critic.
• reason / next_step - the model's actual reply, useful for reading why a grade came out the way it did.
• grader - "deterministic" or "critic", so you know which verdict you're looking at.

Success criteria for shipping a skill:

1. trigger_correct / total == 1.0 (every positive fires, every negative doesn't).
2. Under critic: expectation_pass / expectation_total >= 0.85 with flaws addressed or documented.
3. The NEXT_STEP text for each positive eval names the specific commands/paths/conventions the skill taught.

If (1) fails, the skill's WHEN clause is wrong or weak - rewrite it, don't patch around it. If (2) or (3) fails, the skill's DO list isn't emphatic enough - check if a rule is buried in prose; promote it into a dedicated section with a canonical example.

Iteration workflow

1. Draft or edit the skill.
2. Write/update evals/evals.json (positive + negative per skill, minimum).
3. ai-skill-eval run <skill> and read the markdown report.
4. For each failing eval:
   • Read the NEXT_STEP text carefully - what did the model miss?
   • If the miss is in the skill's wording, revise the skill. Common fixes:
     • Canonical commands that paraphrased away → promote to an explicit code block.
     • Rules that got dropped → split into their own named section.
     • Over-triggering → tighten the WHEN clause with an explicit exclusion.
   • If the miss is genuinely a model limitation unlikely to improve, write a more permissive expectation or document the limitation.
5. ai-skill-eval rerun <skill>:<failed-eval-id> to confirm the fix.
6. Once the affected evals pass, run the full suite once more to catch regressions.

Workspace layout

Default: .ai-skill-eval/ in cwd (gitignored in this repo). Per skill:

.ai-skill-eval/
└── <skill-name>/
    ├── prompts/<eval-id>.txt        (what was sent to the driver)
    ├── results/<eval-id>.txt        (raw model reply)
    └── grades/<eval-id>.json        (parsed + graded)

grade / report / rerun / optimize all read from this workspace. Override with --workspace DIR to keep per-run workspaces separate (handy when benchmarking several models back-to-back).

Description optimization (optimize)

ai-skill-eval optimize <skill> iteratively rewrites the description: frontmatter until the trigger-rate on a held-out test set stops improving. It uses the same driver abstraction as run, so any --driver pi|claude|codex or --driver-cmd '…' works.

Minimum input: a trigger-only eval set at <skill>/evals/trigger-evals.json:

[
  { "query": "…realistic user message…", "should_trigger": true },
  { "query": "…near-miss message…", "should_trigger": false }
]

If trigger-evals.json is absent, the optimizer falls back to <skill>/evals/evals.json and projects each entry to {query: prompt, should_trigger} (expectations are ignored for the loop). Aim for 20 items, 8–10 each side, so the 0.4 default holdout leaves a meaningful train/test split.

Default run: ai-skill-eval optimize <skill> → prints the best-scoring description to stdout. Nothing touches SKILL.md. Pass --write to rewrite the frontmatter in place; the previous description is snapshotted to .ai-skill-eval/<skill>/description-history.json and a unified diff is printed before the overwrite.

Key flags:

• --eval-set PATH - explicit eval-set file.
• --holdout F - stratified test fraction. 0 disables (train = entire eval set).
• --max-iterations N - default 5. Loop exits early when the train set reaches zero failures.
• --runs-per-query N - stochastic trigger measurement (same as run).
• --trigger-threshold T - pass threshold for trigger_rate.

Per-iteration artifacts land under iteration-N/optimize/improver/{prompt,response,parsed}.{txt,json}. Trigger grades and per-run files live under iteration-N/with_skill/ just like run, so report --iteration N --compare-to M works out of the box for comparing two optimizer iterations.

Anti-patterns

• Shipping a skill with only positive evals. One positive per skill tells you it triggers; without a negative you don't know whether it only triggers on the right scenarios.
• Treating a 0/N deterministic expectation score as failure. The default grader is a lower bound - 0/N on well-written skills is common when expectations are natural-language. Run with --critic-cmd before despairing.
• Editing expectations to match the model's reply. If the model got it wrong, the skill is probably wrong - fix the skill. Only edit expectations when the original wording was genuinely unverifiable.
• Running against only one model. Skills that pass qwen3 pass claude-haiku; the reverse isn't guaranteed. Test on the weakest model you support.
• Skipping the rerun step. After a skill revision, a targeted rerun is seconds; a full suite is minutes. Use the right tool.

Quick reference