Paperbench Eval

paperbench-eval

Toward Autonomous Long-Horizon Engineering for ML Research — Chen et al. (2026) (arXiv:2604.13018, 2026)

What this evaluates

Evaluates an autonomous agent's ability to replicate top-tier conference ML papers from scratch. It probes long-horizon engineering capabilities by measuring performance across 20 diverse tasks under a strict 24-hour time and compute budget.

Datasets

• PaperBench — total 20; splits: test (20)

Metrics

• Average Score (primary) — range: percent
  • Arithmetic mean of task-level scores across all 20 tasks. Scores are percentages derived from the official grading protocol.

Input / output format

Input: A top-tier conference ML paper to be replicated from scratch, with a 24-hour time budget and one H20 GPU allocated per task.

Output: Replicated code, experimental setups, and results for the target paper.

Scoring recipe

def score_paperbench(predictions, gold):
    task_scores = []
    for task in predictions:
        # Graded by GPT-5.4 using official PaperBench protocol
        score = grade_with_gpt54(task.output, task.gold)
        task_scores.append(score)
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Grading relies on GPT-5.4, which costs ~$832 for the full 20-task evaluation, severely limiting repeated runs.
• Cost per task varies significantly across baselines (e.g., $27.44 vs $15.67), so efficiency must be tracked alongside accuracy.

Evidence (verbatim from paper)

Table 1: Main results on PaperBench full evaluation. Values in red indicate AiScientist’s gains over the best baseline. Bold and underlined denote the best and second-best results within each LLM.

| Average Score | 19.26 | 20.60 | 30.52 | +9.92 | 22.58 | 22.37 | 33.73 | +11.15 |

Citation

@misc{chen2026toward,
  title={Toward Autonomous Long-Horizon Engineering for ML Research},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.13018}
}

• arXiv: 2604.13018