Mle Bench Lite Eval

mle-bench-lite-eval

Toward Autonomous Long-Horizon Engineering for ML Research — Chen et al. (2026) (arXiv:2604.13018, 2026)

What this evaluates

Evaluates an agent's ability to iteratively refine and improve runnable solutions for competition-style ML tasks over a long horizon. It probes sustained experiment improvement and competitive performance rather than just initial submission validity.

Datasets

• MLE-Bench Lite — total ?; splits: test (-1)

Metrics

• Any Medal% (primary) — range: percent
  • Percentage of tasks where the agent's submission achieves at least a Bronze medal. Also reports ValidSubmission%, AboveMedian%, Bronze%, Silver%, and Gold%.

Input / output format

Input: Competition-style ML tasks requiring iterative experiment improvement, with a 24-hour time budget and one H20 GPU per task.

Output: Runnable solutions/submissions for ML tasks.

Scoring recipe

def score_mle_bench_lite(predictions, gold):
    medal_counts = {'ValidSubmission': 0, 'AboveMedian': 0, 'Bronze': 0, 'Silver': 0, 'Gold': 0, 'AnyMedal': 0}
    for task in predictions:
        grade = grade_competition_submission(task.output, task.gold)
        if grade.is_valid: medal_counts['ValidSubmission'] += 1
        if grade.above_median: medal_counts['AboveMedian'] += 1
        if grade.medal in ['Bronze', 'Silver', 'Gold']: medal_counts[grade.medal] += 1
        if grade.medal != 'None': medal_counts['AnyMedal'] += 1
    total = len(predictions)
    return {k: v/total for k, v in medal_counts.items()}

Common pitfalls

• Official leaderboard results are not directly matched comparisons due to different setups and models, so they should only be used for contextual reference.
• Grading focuses on sustained improvement across multiple rounds, meaning early valid submissions do not guarantee high medal counts.

Evidence (verbatim from paper)

MLE-Bench Lite [Chan et al., 2025] evaluates sustained experiment improvement on top-tier competition-style ML tasks, with Any Medal% as the primary metric. ... On the controlled evaluation, AiScientist delivers the strongest overall performance under both backbones. It reaches the same 81.82 Any Medal% with Gemini-3-Flash and GLM-5, improving over the strongest matched baseline by 4.55 and 18.18 points, respectively.

Citation

@misc{chen2026toward,
  title={Toward Autonomous Long-Horizon Engineering for ML Research},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.13018}
}

• arXiv: 2604.13018