# Choose Agent Benchmarks

> Research, select, audit, reproduce, and design benchmark suites for LLM agents. Use for agent-benchmark surveys, related-work comparisons, evaluation plans, benchmark selection, leaderboard interpretation, reproducibility reviews, or experiments on web agents, computer-use agents, tool/API agents, software-engineering agents, deep-research/scientific agents, memory agents, multi-agent systems, safety/security agents, and embodied agents. Also use when comparing a base model, an agent scaffold/harness, or a complete agent system; when diagnosing saturation, contamination, reward hacking, environment drift, evaluator validity, cost, or reliability; and when studying model–harness co-evolution. Distinguish full interactive benchmarks from static diagnostics, environments, harnesses, and evaluator benchmarks.

- Skill: `pnx2003/choose-agent-benchmarks` (Agent Skill, multi-file: 13 files)
- Install (CLI): `npx skillmds@latest add pnx2003/choose-agent-benchmarks`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pnx2003/choose-agent-benchmarks/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: pnx2003 (https://skillmd.com/u/pnx2003)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/pnx2003/choose-agent-benchmarks

---


# Choose Agent Benchmarks

Build an evaluation portfolio whose evidence matches the research claim. Use the curated atlas, current-version cautions, resource estimates, and audit protocol instead of choosing a benchmark from name recognition or a single leaderboard.

## Start with the Claim

1. State the candidate boundary before naming benchmarks:
   - **Model comparison:** hold scaffold, prompts, tools, environment, budget, and evaluator fixed.
   - **Scaffold comparison:** hold model and environment fixed; vary orchestration, memory, planning, retrieval, or tool routing.
   - **Complete-system comparison:** treat model, scaffold, tools, and policies as one candidate; do not attribute gains to one component.
   - **Robustness claim:** vary environment seeds, task instances, tool failures, latency, observation format, or user behavior.
2. Write the narrowest causal question. Examples: “Does episodic memory improve multi-session retrieval?” and “Does GUI/API routing improve completion under equal action budgets?”
3. Define the deployment analogue: user, environment, action space, horizon, stakes, and acceptable side effects.
4. Record hard constraints: licenses, API access, internet, Docker/VM/GPU availability, runtime, monetary budget, and human-review capacity.

If the claim is unclear, ask for the target agent, task domain, candidate boundary, and budget before recommending a suite.

## Route to the Atlas

Read `references/selection-guide.md` for the taxonomy, decision matrix, validity checks, and reporting contract. Then load only the relevant domain reference:

- Generalist, planning, web/browser, computer-use, or GUI: `references/general-web-computer.md`
- Tool/API/conversational, software engineering, or terminal: `references/tools-coding.md`
- Scientific research, deep research, memory, or multi-agent systems: `references/research-memory-multiagent.md`
- Safety/security or embodied/game agents: `references/safety-embodied.md`
- Current 2025–2026 additions and unstable versions: `references/frontier-watchlist.md`
- Benchmark methodology, cost, disclosure, reward hacking, and statistical design: `references/evaluation-foundations.md`

Use `scripts/search_catalog.py` to filter the compact catalog without loading every profile:

```bash
python scripts/search_catalog.py --domain web --maturity canonical
python scripts/search_catalog.py --needs vm --interaction dynamic
python scripts/search_catalog.py --query "multi-agent security"
python scripts/search_catalog.py --list-values
```

The atlas was source-checked through **2026-08-25**. Browse official paper, repository, dataset card, and benchmark site before making claims about a current release, task count, leaderboard result, license, access policy, or runtime dependency.

## Select a Portfolio

Choose three complementary layers when resources allow:

1. **Core benchmark:** an interactive, domain-valid benchmark that supports the main claim.
2. **Diagnostic benchmark:** a cheaper test isolating the suspected capability, such as function calling, GUI grounding, long-memory retrieval, or evaluator quality.
3. **Frontier or stress benchmark:** a dynamic, long-horizon, adversarial, multimodal, or asynchronous benchmark that tests whether the conclusion survives harder conditions.

Prefer coverage across these axes rather than many near-duplicates:

- Static dataset vs stateful simulator vs live/asynchronous environment
- Text/API vs DOM vs screenshot/mouse-keyboard vs terminal vs multimodal/voice
- Short horizon vs long horizon; single app vs cross-app; single agent vs multi-agent
- Exact answer vs state transition vs tests vs checkpoints vs human/LLM judge
- Reproducible local environment vs live-web realism
- Benign utility vs side effects, attacks, or policy compliance

Do not average unrelated domains into one number unless the weighting has a deployment interpretation. Report per-domain and per-capability results first.

## Audit Before the Main Run

Run a small pilot and challenge the measurement, not only the agent:

1. **Pin identity:** benchmark release, task manifest, repository commit, container/VM image, website snapshot, model version, harness commit, prompts, tools, and evaluator version.
2. **Test the evaluator:** run a no-op agent, known-success trajectory, known-failure trajectory, malformed output, and at least one reward-hacking attempt.
3. **Inspect tasks:** manually sample instructions, initial states, hidden constraints, tests, reference answers, and teardown/reset behavior.
4. **Check leakage:** search public training-like artifacts and compare performance by task or repository age where feasible.
5. **Check environment stability:** repeat identical trajectories; log resets, CAPTCHA, timeouts, API drift, nondeterministic state, and unavailable assets.
6. **Check judge validity:** blind a stratified sample to humans; measure agreement, false-positive success, and false-negative success.
7. **Check cost fairness:** normalize or stratify by tokens, tool calls, wall time, retries, parallelism, model/API price, and compute.

Treat a failed audit as a limitation or redesign trigger. Do not silently remove failed tasks after seeing system outcomes.

## Run and Report

For stochastic agents, run repeated trials per task. Report task-level success probabilities or confidence intervals; include pass@1 and a reliability measure such as pass^k when repeated success matters. Do not substitute best-of-k for reliability.

Always report:

- Candidate boundary and controlled variables
- Benchmark release, split, task count actually run, exclusions, and reasons
- Observation/action interface and allowed tools
- Model, scaffold, prompts, context policy, memory, retrieval, and tool routing
- Step, token, time, retry, parallelism, and monetary budgets
- Success metric plus partial progress, side effects, safety, latency, and cost where relevant
- Number of trials, seeds, uncertainty, and failure taxonomy
- Environment failures separated from agent failures
- Evaluator audit and human-validation sample
- Per-task results or a machine-readable result artifact when permitted

Never compare published scores unless benchmark version, split, harness, environment, evaluator, candidate boundary, and budget are compatible. If they differ, call the comparison directional rather than head-to-head.

## Handle High-Risk Benchmark Families

- Treat **SWE-bench Verified** and **SWE-bench Pro** as historically important but currently high-risk sources for sole frontier claims. Use audited subsets, disclose broken/contaminated tasks, and triangulate with Terminal-Bench, SWE-Lancer, FeatureBench, or a private temporal holdout.
- Pin **Terminal-Bench** to 2.1 or a precise 3.x manifest; do not mix 2.0, 2.1, and 3.x scores.
- Pin **BFCL** to a version and category. Function-call correctness is not equivalent to end-to-end agent success.
- Use the current corrected **tau** implementation or verified fork; name the family member and domain. Do not collapse tau-bench, tau2, tau3, and voice scores.
- For **live-web** benchmarks, time-stamp every run and treat website/search changes as part of the experimental condition.
- For **OSWorld** and other VM benchmarks, pin image and asset versions; separate website outages and reset failures from agent mistakes.
- Treat **ScreenSpot-Pro**, **Mind2Web**, **LongMemEval**, and similar static tests as diagnostics, not complete agent evaluations.
- Treat **BrowserGym**, **Harbor**, **ARE**, and **SWE-Universe** primarily as infrastructure or environment layers unless the named task set is also specified.

## Design Model–Harness Co-evolution Studies

Use a factorial experiment instead of a single leaderboard:

1. Cross at least two models with at least two harnesses.
2. Hold tools, environment, prompts, evaluator, and budget fixed within each cell.
3. Add ablations for planning, memory, retrieval, reflection, and tool routing.
4. Test scaffold transfer to an unseen model and model transfer to an unseen scaffold.
5. Add environment perturbations and repeated trials.
6. Report main effects, interaction effects, cost-normalized performance, and failure-mode shifts.

A benchmark gain that appears only for one model–harness pair is an interaction, not a general scaffold improvement.

## Produce the Recommendation

Return a compact evidence table with these columns:

| Benchmark | Role | Claim coverage | Interface/environment | Metric | Resource tier | Principal risk | Why included |
|---|---|---|---|---|---|---|---|

Then provide:

1. The recommended minimal suite and expanded suite
2. A run order from cheapest diagnostic to most expensive interactive test
3. Version pins and reproducibility requirements
4. Metrics, repetitions, and statistical analysis
5. Expected confounds and an audit checklist
6. A clear list of claims the suite cannot support

Use primary sources for factual claims. Mark uncertain, version-dependent, or inferred information explicitly.
Answer in the user's language unless they request another language.