Agent Benchmark

Agent Library Benchmark

Compares two configurations of the agent library (typically old = snapshot + previous models, new = current defs + current models) on a fixed eval set, with objective assertions and honest cost/latency accounting.

Layout

Workspace: ~/.claude/agents-improvement-workspace/

• evals/evals.json — canonical eval set (prompts + assertions). Extend here, not ad hoc.
• agents-snapshot-*/ — frozen agent definitions used as old baselines.
• iteration-N/ — one benchmark run: eval-<name>/<config>/run-1/{outputs/deliverable.md, timing.json, grading.json} plus benchmark.json, benchmark.md, review.html.

Workflow

1. Snapshot before editing. cp -r ~/.claude/agents <workspace>/agents-snapshot-<label> — the old config must be reproducible. Note: ~/.claude/agents and ~/.claude/skills are symlinks into the claude-dev-agents repo, so edits land there.
2. Pick models per config. Old runs pin the previous model IDs (verify they are still served via the claude-api skill's model catalog — never guess IDs). New runs use the agents' current frontmatter models.
3. Run both configs in parallel with scripts/run_eval.sh (headless claude -p; the agent file's body becomes --append-system-prompt-file; deliverable is stdout; timing/cost captured from --output-format json). Run against the real target repo so agents can ground themselves — grounded evals discriminate far better than abstract ones.
4. Grade with subagents. One grader per eval grading BOTH configs (consistent bar). Graders follow skill-creator/agents/grader.md and write grading.json with expectations: [{text, passed, evidence}] plus a summary: {passed, failed, total, pass_rate} block (the aggregator needs summary). Ask each grader to also critique assertion discrimination — the most valuable output is often "these assertions don't separate the configs; here are sharper ones."
5. Aggregate + analyze. cd ~/.claude/skills/skill-creator && python3 -m scripts.aggregate_benchmark <iteration-dir> --skill-name <name>. Then add analyst_observations to benchmark.json covering: non-discriminating assertions, quality differences no assertion captures, cost/token/latency deltas (report totals honestly — newer models often spend more exploration tokens), and any behavior finding that implies an agent-definition fix.
6. Report. python3 ~/.claude/skills/skill-creator/eval-viewer/generate_review.py <iteration-dir> --skill-name <name> --benchmark <iteration-dir>/benchmark.json --previous-workspace <prev-iteration> --static <iteration-dir>/review.html (static output — WSL2 has no browser auto-open).
7. Close the loop. Every behavioral failure should become either an agent-definition fix (validate with a targeted re-run in a new iteration dir) or a sharper assertion for the next iteration. Score gaps are only meaningful once you've separated "worse artifact" from "different behavior" — e.g. an agent that asks a clarifying question instead of delivering scores 0 but may need a definition fix, not a model downgrade.

Skill trigger benchmark

scripts/run_trigger_benchmark.py [--model claude-sonnet-5] [--only skill-a,skill-b] [--out results.json] — for every skill with evals/trigger-eval.json (17 skills, ~268 queries), a judge model routes each query against the REAL roster of all skill frontmatter descriptions and picks one skill or NONE. Scores per-skill recall (should-trigger routed home) and false-fire rate. Re-run after editing any skill description or when the session model changes (the session model makes routing decisions, so a model upgrade can shift triggering).

Reading results: false-fires are the expensive failure (wrong skill loads); under-triggers routed to a sibling skill in the same cluster (jira-reader/writer, spec-plan/review, scope-question/research-gated) are usually acceptable — don't chase them by stuffing descriptions. Under-triggers routed to NONE are description gaps: add generalized trigger patterns to the description (never the eval queries verbatim — that's overfitting), then validate with --only <skill>. Baseline (2026-07-30, claude-sonnet-5): 92.4% overall, zero false-fires; after description fixes on the 3 worst skills, their recall rose 0.375→0.75, 0.5→0.7, 0.625→0.875.

Known pitfalls

• Headless runs auto-deny permission prompts: evals must be deliverable-as-text (design docs, diagnoses, plans), not file-writing tasks.
• Headless runs are also permission-scoped to the run's cwd (7th arg to run_eval.sh): any fixture file an eval references must live UNDER that cwd, or every read is denied and the run silently produces a blocked-explanation instead of a deliverable. Check deliverable sizes after a batch — a run under ~1.5KB usually means blocked tool access, not a terse answer.
• Baseline contamination (with/without-skill outcome runs): a plain headless run still loads the ENTIRE user skill library from ~/.claude/skills, so a "without_skill" baseline can trigger or read the very skill under test — graders catch it as skill-internal vocabulary appearing verbatim in baseline output. Fix: run baselines with env CLAUDE_CONFIG_DIR=<bare-dir> where the bare dir contains only .credentials.json and .claude.json copied from ~/.claude (CLI-bundled skills remain; the user library disappears). Also keep fixtures OUTSIDE any skill directory — a baseline agent browsing next to a fixture can discover SKILL.md two levels up.
• Outcome runs are n=1 per config by default — treat close scores as ties, and re-run before trusting a small delta. Empty assertions arrays force graders to derive expectations each time; persist derived expectations back into eval_metadata.json for consistency across iterations.
• An agent that ends with only a clarifying question produces a useless run — agent definitions carry a "Working Non-Interactively" section for this; if a run still does it, fix the definition before blaming the model.
• Assertions that any competent output satisfies measure nothing. Prefer assertions tied to the real repo (does the diagnosis land on the code path that actually renders? does the plan surface the known policy conflict?).
• Compare cost as totals AND per-token rates; a cheaper model that explores more can cost the same per run.