Eval regression
Use deterministic repository tests first. Stop when the target has no behavioral diff.
Resolve the plugin from the argument or cwd. Its catalog is evals/evals.json; the shared runner is ../ai-agent-bench/scripts/run_evals.py. Require the user to choose agent, model, and effort. Never select a costly model or high effort silently.
Normal check
- Select cases by changed paths with
--changed-from <base>, or name them with --case. Do not select the full catalog implicitly.
- Run the command without
--run. The runner prints the cases, modes, repeat count, session count, per-session timeout, and maximum duration without starting an agent.
- Present that plan and stop for explicit cost approval.
- After approval, repeat the same command with
--run. The default is candidate-only, one run per case, 180 seconds per session, and at most four sessions.
- Report every failed assertion, timeout, non-zero exit, duration, and token count. Missing evidence is inconclusive.
Escalation
- Compare base and candidate only when the user asks, or when a failed candidate check needs to distinguish a regression from an existing failure. Extract the base with
git archive; run the same selected cases, agent, model, effort, repeat, and timeout on both; compare reports with --compare.
- Repeat only a failed or observably unstable case. Three repeats are a stability benchmark, not a default.
--all, --repeat > 1, a larger --max-sessions, or a longer timeout needs a new run plan and explicit approval.
- Routing cases stop at the first Skill selection. A tool assertion is appropriate there because routing is the contract; they must not execute the selected skill.
Remove temporary base copies after the comparison. Do not edit or commit the target.
Deterministic assertions are tool, tool_not, clean_worktree, changed_files_exact, file_contains, file_not_contains, transcript_contains, and tool_sequence. Use a semantic judge only when no filesystem, command, tool, ordering, or assistant-output observation can express the contract.
Source: hashgraph-online/awesome-codex-plugins → plugins/reidemeister94/development-skills/skills/eval-regression/SKILL.md
1---
2name: eval-regression
3description: "Use when the user asks to check a plugin or skill with behavioral evals, compare it with HEAD, or investigate a behavioral regression."
4allowed-tools: "Glob, Read, Bash, Grep, AskUserQuestion"
5category: testing-and-qa
6source_repo: hashgraph-online/awesome-codex-plugins
7source_path: "plugins/reidemeister94/development-skills/skills/eval-regression/SKILL.md"
8source_url: https://github.com/hashgraph-online/awesome-codex-plugins/blob/HEAD/plugins/reidemeister94/development-skills/skills/eval-regression/SKILL.md
9metadata:
10 source: https://skillmd.com/skills/hashgraph-online/eval-regression
11---
12
13
14# Eval regression
15
16Use deterministic repository tests first. Stop when the target has no behavioral diff.
17
18Resolve the plugin from the argument or cwd. Its catalog is `evals/evals.json`; the shared runner is `../ai-agent-bench/scripts/run_evals.py`. Require the user to choose agent, model, and effort. Never select a costly model or high effort silently.
19
20## Normal check
21
221. Select cases by changed paths with `--changed-from <base>`, or name them with `--case`. Do not select the full catalog implicitly.
232. Run the command without `--run`. The runner prints the cases, modes, repeat count, session count, per-session timeout, and maximum duration without starting an agent.
243. Present that plan and stop for explicit cost approval.
254. After approval, repeat the same command with `--run`. The default is candidate-only, one run per case, 180 seconds per session, and at most four sessions.
265. Report every failed assertion, timeout, non-zero exit, duration, and token count. Missing evidence is inconclusive.
27
28## Escalation
29
30- Compare base and candidate only when the user asks, or when a failed candidate check needs to distinguish a regression from an existing failure. Extract the base with `git archive`; run the same selected cases, agent, model, effort, repeat, and timeout on both; compare reports with `--compare`.
31- Repeat only a failed or observably unstable case. Three repeats are a stability benchmark, not a default.
32- `--all`, `--repeat > 1`, a larger `--max-sessions`, or a longer timeout needs a new run plan and explicit approval.
33- Routing cases stop at the first Skill selection. A tool assertion is appropriate there because routing is the contract; they must not execute the selected skill.
34
35Remove temporary base copies after the comparison. Do not edit or commit the target.
36
37Deterministic assertions are `tool`, `tool_not`, `clean_worktree`, `changed_files_exact`, `file_contains`, `file_not_contains`, `transcript_contains`, and `tool_sequence`. Use a semantic judge only when no filesystem, command, tool, ordering, or assistant-output observation can express the contract.
38
39---
40
41**Source:** [`hashgraph-online/awesome-codex-plugins`](https://github.com/hashgraph-online/awesome-codex-plugins) → `plugins/reidemeister94/development-skills/skills/eval-regression/SKILL.md`