Exp Bench Eval

exp-bench-eval

EXP-Bench: Can AI Conduct AI Research Experiments? — Kon et al. (2025) (arXiv:2505.24785, 2025)

What this evaluates

Evaluates AI agents' end-to-end capability to conduct real AI research experiments, including designing methodologies, implementing code, executing experiments, and drawing conclusions.

Datasets

• EXP-Bench — total 461; splits: test (461); repo https://github.com/Just-Curieous/Curie

Metrics

• All·E✓ (primary) — range: percent
  • Conjunctive metric requiring full correctness in design (D), implementation (I), conclusion (C), and executability (E). Score is the product of the four component scores.
• D — range: percent
  • Design correctness, defined as the proportion of design criteria met by the agent's experimental plan.
• I — range: percent
  • Implementation correctness, defined as the proportion of implementation components satisfied by the agent's code.
• C — range: percent
  • Conclusion correctness, assessing whether the agent's analytical conclusions align with ground truth.
• E — range: percent
  • Executability, a binary metric indicating whether the agent-generated code runs successfully in a clean containerized environment and produces expected outputs.
• I·E — range: percent
  • Conjunctive metric combining implementation correctness and executability.
• All✓ — range: percent
  • Tasks fully correct in terms of D, I, and C.
• M — range: percent
  • Integrity check monitoring for disallowed behaviors (accessing PDF, git operations, or using fake data).

Input / output format

Input: Masked GitHub repository of a target AI paper (with task-specific scripts removed), experimental instructions, and relevant context (e.g., API credentials), executed in an Ubuntu 24.04 Docker container with 4× Nvidia A40 GPUs.

Output: Agent-generated code modifications, experimental design documents, implementation scripts, and analytical conclusions.

Scoring recipe

def evaluate(task, agent_output, gt):
    m = 0 if (accessed_pdf or git_ops or fake_data) else 1
    if m == 0: return 0.0
    d = proportion_met(agent_output.design, gt.design)
    i = proportion_met(agent_output.code, gt.code)
    c = 1.0 if conclusion_matches(agent_output.conclusion, gt.conclusion) else 0.0
    e = 1.0 if run_in_clean_container(agent_output.code) and produces_expected_output() else 0.0
    return {
        'M': m, 'D': d, 'I': i, 'C': c, 'E': e,
        'I·E': i * e,
        'All✓': d * i * c,
        'All·E✓': d * i * c * e
    }

Common pitfalls

• Agents can produce plausible but unfounded conclusions (C) or mock implementations that successfully execute (E), causing overestimation bias in partial metrics.
• Execution checking is computationally expensive, so only a subset of traces are run through the Code Execution Validator (#E), which may skew results if early failures are automatically discarded.
• Conjunctive metrics (e.g., All·E✓) drop scores drastically compared to partial metrics, masking agent capabilities if only strict end-to-end scores are reported.

Evidence (verbatim from paper)

These assessments yield scores for: D (design correctness, i.e., proportion of design criteria met), I (implementation correctness, i.e., proportion of implementation components satisfied), and C (conclusion correctness). This executability metric is denoted as E. All✓ denotes tasks that are fully correct in terms of D, I, and C, while All·E✓ adds the executability requirement.

Citation

@misc{kon2025expbench,
  title={EXP-Bench: Can AI Conduct AI Research Experiments?},
  author={Kon et al. (2025)},
  year={2025},
  note={arXiv:2505.24785}
}

• arXiv: 2505.24785