Evaluate Skill

Evaluate Skill

Run a skill repeatedly to measure how reliably it works, and design the evals that measure it.

Prerequisites

The caliper CLI must be on PATH. This skill can be copied into an agent without the Caliper repo, so do not assume the CLI is packaged with it. Install if missing:

pipx install caliper-eval

The engine (backend + model) is not part of the spec — it is chosen at run time with --model (skill) and --judge-model (judge), independently, from claude-code, codex, pi, defaulting to claude-code. Every backend is a CLI agent that uses its own subscription/auth; there is no direct-API backend (for API billing, configure a CLI with an API key). Full per-backend detail and every command: REFERENCE.md.

Spec shape

An .eval.yaml names the skill and a list of tasks. Keep skill.path relative to the spec file (usually ./SKILL.md):

skill:
  path: ./SKILL.md      # relative to the spec file
tasks:
  - name: What success looks like
    prompt: <prompt sent to the skill under test>
    expect: <natural-language pass/fail criterion>
    assert: |           # optional deterministic Python check
      assert ...

The spec has no backend/model or judge: block; pick the engine when you run, e.g. caliper run <spec> --model codex --judge-model codex. The full format (setup/cleanup, external assert scripts, sandbox) is in REFERENCE.md.

Bundled references

references/evals/ holds complete real examples (Claude Code smoke, commit workflow, screenshot, summarization, TDD) — each folder self-contained with its fixture SKILL.md and .eval.yaml. references/simple.eval.yaml is one compact multi-task spec.

No eval yet?

If the skill has a SKILL.md but no .eval.yaml, suggest the grill-skill workflow — it interviews the user and generates a happy/edge/adversarial spec. Use evaluate-skill directly when a spec already exists and the user wants to run, validate, report, or extend it.

Designing good evals

1. Name the target behavior — what should the skill do better than the base agent?
2. Decide whether the suite is a capability eval or a regression eval.
3. Cover normal, edge, and adversarial cases when the behavior matters.
4. Grade artifacts (files, git state, command output, exact values) whenever you can; judge the transcript only when the behavior itself is the point. The full artifact-vs-transcript rules, the task-quality checklist, common eval patterns, and how to write expect: rubrics live in REFERENCE.md — read and apply them when designing tasks.
5. Run once with --ablate <skill-name> and caliper compare the two runs, to confirm the skill beats the raw agent. Debug the spec at --k 1, then measure reliability at --k 3 or higher.

Done when: tasks have observable success criteria, at least one deterministic assert:, a positive delta against the ablated run, the spec passes caliper validate, and the user has been prompted to commit the spec.

Committing

Running Caliper produces two artifacts: the .eval.yaml spec — the valuable one, commit it beside the skill so anyone who clones the repo can run the same eval — and .caliper/results/ saved run JSONs, useful for diffing over time and safe to gitignore. After creating or running an eval, tell the user to commit the spec alongside SKILL.md.