How do I install the Evals skill?

Run npx skillmds@latest add compoundingtech/evals in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Evals safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Evals?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Evals free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

compoundingtech (@compoundingtech) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Evals by compoundingtech — Agent Skill

Name: Evals
Author: compoundingtech

File contents

evals

Use the repository's canonical surfaces:

AGENT-SPEC.md for current hand-authored st2 agent declarations;
CATALOG.md for the generated inventory, cost bands, exclusions, and evidence;
cells/<cell>/<cell>.kdl for executable cell behavior.

Default workflow

Run the free gate first:

bin/check-corpus.sh

Inspect the no-execution overnight plan with:

bin/overnight.sh --dry-run
bin/overnight.sh --dry-run --cell <cell> --cell <cell>

Never infer permission for a paid/model run. When one cell is explicitly authorized:

st2 eval ./cells/<cell>/ --keep

When an exact conservative paid queue is explicitly authorized, repeat --cell in the approved order:

bin/overnight.sh --run --cell <cell> --cell <cell> --state-dir .eval-runs/overnight

The common Codex reset-available banner stops the default run before the next cell. A future full unattended run therefore requires separate human review of both --all and the higher-spend informational-banner opt-in:

bin/overnight.sh --run --all --allow-informational-reset-banner --state-dir .eval-runs/overnight

Paid --run rejects an omitted selector, duplicate, unknown, or retired cell. Explicitly selected maintained model-free cells execute through the same st2 lifecycle and receipts without provider checks; Claude/Codex binary and authentication checks apply only when the selected subset needs that provider. Hard quota/rate-limit errors stop unconditionally in both modes. Never infer approval for either paid command.

Cell rules

One canonical root KDL per cell and one declared max-timeout.
The eval runner owns the native flat bus and pty roots.
Every model subject and model judge selects its model and medium effort explicitly.
Every bus-connected model agent uses native bare ding, one cold-start drain, and event-first DING wakeups.
Every Claude workspace materializes the canonical Claude loader/persona/hooks; every Codex workspace materializes the canonical AGENTS/hooks and trusts them at launch.
Fixtures are synthetic and resettable; checked-in Git metadata is frozen as _git.
Graders are held out, ownership-aware, and mutation-valid where practical.
No paid rerun is a substitute for shell, parse, reset, model-policy, and PII gates.

After changing inventory or structured evidence, run:

bin/generate-catalog.sh --write
bin/check-corpus.sh

---name: evalsdescription: Inspect, validate, or run isolation-gated st2 agent-team eval cells. Use for corpus inventory, free preflight, an explicitly authorized cell run, or adding a held-out-graded cell.---# evalsUse the repository's canonical surfaces:- `AGENT-SPEC.md` for current hand-authored st2 agent declarations;- `CATALOG.md` for the generated inventory, cost bands, exclusions, and evidence;- `cells/<cell>/<cell>.kdl` for executable cell behavior.## Default workflowRun the free gate first:```shbin/check-corpus.sh```Inspect the no-execution overnight plan with:```shbin/overnight.sh --dry-runbin/overnight.sh --dry-run --cell <cell> --cell <cell>```Never infer permission for a paid/model run. When one cell is explicitly authorized:```shst2 eval ./cells/<cell>/ --keep```When an exact conservative paid queue is explicitly authorized, repeat `--cell` in the approved order:```shbin/overnight.sh --run --cell <cell> --cell <cell> --state-dir .eval-runs/overnight```The common Codex reset-available banner stops the default run before the next cell. A future full unattendedrun therefore requires separate human review of both `--all` and the higher-spend informational-banner opt-in:```shbin/overnight.sh --run --all --allow-informational-reset-banner --state-dir .eval-runs/overnight```Paid `--run` rejects an omitted selector, duplicate, unknown, or retired cell. Explicitly selected maintainedmodel-free cells execute through the same st2 lifecycle and receipts without provider checks; Claude/Codexbinary and authentication checks apply only when the selected subset needs that provider. Hard quota/rate-limiterrors stop unconditionally in both modes. Never infer approval for either paid command.## Cell rules- One canonical root KDL per cell and one declared `max-timeout`.- The eval runner owns the native flat bus and pty roots.- Every model subject and model judge selects its model and medium effort explicitly.- Every bus-connected model agent uses native bare `ding`, one cold-start drain, and event-first DING wakeups.- Every Claude workspace materializes the canonical Claude loader/persona/hooks; every Codex workspace materializes the canonical AGENTS/hooks and trusts them at launch.- Fixtures are synthetic and resettable; checked-in Git metadata is frozen as `_git`.- Graders are held out, ownership-aware, and mutation-valid where practical.- No paid rerun is a substitute for shell, parse, reset, model-policy, and PII gates.After changing inventory or structured evidence, run:```shbin/generate-catalog.sh --writebin/check-corpus.sh```

Frequently asked questions

Evals