Rexbench Eval

rexbench-eval

RExBench: Can coding agents autonomously implement AI research extensions? — Edwards et al. (2025) (arXiv:2506.22598, 2025)

What this evaluates

This benchmark evaluates the ability of LLM-based coding agents to autonomously implement research extensions by modifying existing AI/ML codebases based on domain-expert instructions. It probes complex, multi-step software engineering capabilities, including codebase navigation, hypothesis-driven implementation, and producing executable patches.

Datasets

• REXBench — total 12; splits: test (12); repo https://github.com/tinlaboratory/RexBench

Metrics

• final success rate (primary) — range: [0, 1]
  • Percentage of successful task executions out of total attempts. Each task is run three times to account for agent randomness, and a run is successful if the generated git patch applies and executes correctly in an isolated environment.

Input / output format

Input: Existing codebase, research paper, and domain-expert-written instructions for a specific modification task.

Output: Git patch files containing the proposed code modifications.

Scoring recipe

total_runs = 0
successful_runs = 0
for task in tasks:
    patch = agent.generate(task.inputs)
    for _ in range(3):
        total_runs += 1
        if apply_patch_and_execute(patch, isolated_env):
            successful_runs += 1
return successful_runs / total_runs

Common pitfalls

• Agents often achieve high file recall by correctly locating edit targets but still fail to produce executable code.
• Hints are cumulative; second-level hints do not consistently yield additional benefits over first-level hints.
• Evaluation requires running each task three times to account for agent randomness, which significantly impacts reported success rates.
• Python execution is disabled during the agent's generation phase, but patches are evaluated post-hoc via automated execution.

Evidence (verbatim from paper)

Our main results are shown in Figure 2. We see that most agents struggle with the task, with the best performing agents (OpenHands + Claude 3.7 Sonnet and Claude Code) achieving 25% average final success rates.

Citation

@misc{edwards2025rexbench,
  title={RExBench: Can coding agents autonomously implement AI research extensions?},
  author={Edwards et al. (2025)},
  year={2025},
  note={arXiv:2506.22598}
}

• arXiv: 2506.22598