Choose Agent Benchmarks

Choose Agent Benchmarks

Build an evaluation portfolio whose evidence matches the research claim. Use the curated atlas, current-version cautions, resource estimates, and audit protocol instead of choosing a benchmark from name recognition or a single leaderboard.

Start with the Claim

1. State the candidate boundary before naming benchmarks:
   • Model comparison: hold scaffold, prompts, tools, environment, budget, and evaluator fixed.
   • Scaffold comparison: hold model and environment fixed; vary orchestration, memory, planning, retrieval, or tool routing.
   • Complete-system comparison: treat model, scaffold, tools, and policies as one candidate; do not attribute gains to one component.
   • Robustness claim: vary environment seeds, task instances, tool failures, latency, observation format, or user behavior.
2. Write the narrowest causal question. Examples: “Does episodic memory improve multi-session retrieval?” and “Does GUI/API routing improve completion under equal action budgets?”
3. Define the deployment analogue: user, environment, action space, horizon, stakes, and acceptable side effects.
4. Record hard constraints: licenses, API access, internet, Docker/VM/GPU availability, runtime, monetary budget, and human-review capacity.

If the claim is unclear, ask for the target agent, task domain, candidate boundary, and budget before recommending a suite.

Route to the Atlas

Read references/selection-guide.md for the taxonomy, decision matrix, validity checks, and reporting contract. Then load only the relevant domain reference:

• Generalist, planning, web/browser, computer-use, or GUI: references/general-web-computer.md
• Tool/API/conversational, software engineering, or terminal: references/tools-coding.md
• Scientific research, deep research, memory, or multi-agent systems: references/research-memory-multiagent.md
• Safety/security or embodied/game agents: references/safety-embodied.md
• Current 2025–2026 additions and unstable versions: references/frontier-watchlist.md
• Benchmark methodology, cost, disclosure, reward hacking, and statistical design: references/evaluation-foundations.md

Use scripts/search_catalog.py to filter the compact catalog without loading every profile:

python scripts/search_catalog.py --domain web --maturity canonical
python scripts/search_catalog.py --needs vm --interaction dynamic
python scripts/search_catalog.py --query "multi-agent security"
python scripts/search_catalog.py --list-values

The atlas was source-checked through 2026-08-25. Browse official paper, repository, dataset card, and benchmark site before making claims about a current release, task count, leaderboard result, license, access policy, or runtime dependency.

Select a Portfolio

Choose three complementary layers when resources allow:

1. Core benchmark: an interactive, domain-valid benchmark that supports the main claim.
2. Diagnostic benchmark: a cheaper test isolating the suspected capability, such as function calling, GUI grounding, long-memory retrieval, or evaluator quality.
3. Frontier or stress benchmark: a dynamic, long-horizon, adversarial, multimodal, or asynchronous benchmark that tests whether the conclusion survives harder conditions.

Prefer coverage across these axes rather than many near-duplicates:

• Static dataset vs stateful simulator vs live/asynchronous environment
• Text/API vs DOM vs screenshot/mouse-keyboard vs terminal vs multimodal/voice
• Short horizon vs long horizon; single app vs cross-app; single agent vs multi-agent
• Exact answer vs state transition vs tests vs checkpoints vs human/LLM judge
• Reproducible local environment vs live-web realism
• Benign utility vs side effects, attacks, or policy compliance

Do not average unrelated domains into one number unless the weighting has a deployment interpretation. Report per-domain and per-capability results first.

Audit Before the Main Run

Run a small pilot and challenge the measurement, not only the agent:

1. Pin identity: benchmark release, task manifest, repository commit, container/VM image, website snapshot, model version, harness commit, prompts, tools, and evaluator version.
2. Test the evaluator: run a no-op agent, known-success trajectory, known-failure trajectory, malformed output, and at least one reward-hacking attempt.
3. Inspect tasks: manually sample instructions, initial states, hidden constraints, tests, reference answers, and teardown/reset behavior.
4. Check leakage: search public training-like artifacts and compare performance by task or repository age where feasible.
5. Check environment stability: repeat identical trajectories; log resets, CAPTCHA, timeouts, API drift, nondeterministic state, and unavailable assets.
6. Check judge validity: blind a stratified sample to humans; measure agreement, false-positive success, and false-negative success.
7. Check cost fairness: normalize or stratify by tokens, tool calls, wall time, retries, parallelism, model/API price, and compute.

Treat a failed audit as a limitation or redesign trigger. Do not silently remove failed tasks after seeing system outcomes.

Run and Report

For stochastic agents, run repeated trials per task. Report task-level success probabilities or confidence intervals; include pass@1 and a reliability measure such as pass^k when repeated success matters. Do not substitute best-of-k for reliability.

Always report:

• Candidate boundary and controlled variables
• Benchmark release, split, task count actually run, exclusions, and reasons
• Observation/action interface and allowed tools
• Model, scaffold, prompts, context policy, memory, retrieval, and tool routing
• Step, token, time, retry, parallelism, and monetary budgets
• Success metric plus partial progress, side effects, safety, latency, and cost where relevant
• Number of trials, seeds, uncertainty, and failure taxonomy
• Environment failures separated from agent failures
• Evaluator audit and human-validation sample
• Per-task results or a machine-readable result artifact when permitted

Never compare published scores unless benchmark version, split, harness, environment, evaluator, candidate boundary, and budget are compatible. If they differ, call the comparison directional rather than head-to-head.

Handle High-Risk Benchmark Families

• Treat SWE-bench Verified and SWE-bench Pro as historically important but currently high-risk sources for sole frontier claims. Use audited subsets, disclose broken/contaminated tasks, and triangulate with Terminal-Bench, SWE-Lancer, FeatureBench, or a private temporal holdout.
• Pin Terminal-Bench to 2.1 or a precise 3.x manifest; do not mix 2.0, 2.1, and 3.x scores.
• Pin BFCL to a version and category. Function-call correctness is not equivalent to end-to-end agent success.
• Use the current corrected tau implementation or verified fork; name the family member and domain. Do not collapse tau-bench, tau2, tau3, and voice scores.
• For live-web benchmarks, time-stamp every run and treat website/search changes as part of the experimental condition.
• For OSWorld and other VM benchmarks, pin image and asset versions; separate website outages and reset failures from agent mistakes.
• Treat ScreenSpot-Pro, Mind2Web, LongMemEval, and similar static tests as diagnostics, not complete agent evaluations.
• Treat BrowserGym, Harbor, ARE, and SWE-Universe primarily as infrastructure or environment layers unless the named task set is also specified.

Design Model–Harness Co-evolution Studies

Use a factorial experiment instead of a single leaderboard:

1. Cross at least two models with at least two harnesses.
2. Hold tools, environment, prompts, evaluator, and budget fixed within each cell.
3. Add ablations for planning, memory, retrieval, reflection, and tool routing.
4. Test scaffold transfer to an unseen model and model transfer to an unseen scaffold.
5. Add environment perturbations and repeated trials.
6. Report main effects, interaction effects, cost-normalized performance, and failure-mode shifts.

A benchmark gain that appears only for one model–harness pair is an interaction, not a general scaffold improvement.

Produce the Recommendation

Return a compact evidence table with these columns:

Then provide:

1. The recommended minimal suite and expanded suite
2. A run order from cheapest diagnostic to most expensive interactive test
3. Version pins and reproducibility requirements
4. Metrics, repetitions, and statistical analysis
5. Expected confounds and an audit checklist
6. A clear list of claims the suite cannot support

Use primary sources for factual claims. Mark uncertain, version-dependent, or inferred information explicitly. Answer in the user's language unless they request another language.