thinceller Verify Behavior
Run the behavior evals in evals/ to check that plugin skills and hooks produce the intended behavior in headless Claude Code sessions, and report the results.
Steps
Run
evals/run.sh— all scenarios by default, or only the scenario names passed as$ARGUMENTS. Read the output and report each scenario's PASS/FAIL and trial breakdown.For any scenario that FAILs, re-run it with
KEEP_WORK=1so the workdir and output file are preserved, then read them to determine whether the failure is a plugin regression or a stale/flaky scenario.If the change being verified introduces new behavior not covered by an existing scenario, design and trial a new scenario for it, and add the ones that work reliably to
evals/scenarios/(ascenario.conf+check.shpair). Prefer mechanical checks (git status, grep on output, functional tests on files) over judgment calls; use TRIALS/PASS_MIN to absorb the model's inherent variance.
Cost note
A full run spawns roughly 10 headless claude -p sessions and costs API tokens. Only run this after changes that affect plugin behavior — for structural-only changes, use the verify skill instead.