Mlgym Eval

mlgym-eval

MLGym: A New Framework and Benchmark for Advancing AI Research Agents — Nathani et al. (2025) (arXiv:2502.14499, 2025)

What this evaluates

Evaluates LLM agents on open-ended AI research tasks across 13 diverse benchmarks. It measures the agent's ability to navigate codebases, run experiments, and improve model performance, assessing capabilities from reproducing existing research to achieving state-of-the-art results.

Datasets

• MLGym Benchmarks — total ?; splits: test (-1); repo https://github.com/facebookresearch/MLGym

Metrics

• AutoML-inspired optimization metric (primary) — range: other
  • Quantifies performance improvement across diverse tasks to enable fair cross-task comparison, drawing from AutoML and optimization literature conventions.

Input / output format

Input: Task description, dataset description, ACI documentation, and lightweight generic instructions to act as an ML researcher.

Output: A sequence of up to 50 single-line commands (e.g., file navigation, editing, search, validation, submission) executed in a non-interactive environment.

Scoring recipe

def evaluate_agent(task, agent):
    steps = 0
    max_steps = 50
    context = []
    while steps < max_steps:
        action = agent.step(task, context)
        obs = env.execute(action, timeout=task_timeout)
        context.append((action, obs))
        if action == 'submit':
            break
        steps += 1
    final_codebase = env.get_state()
    score = validate_and_submit(final_codebase, task)
    return score

Common pitfalls

• Agent runs are hard-capped at 50 steps; the final state is autosubmitted if the limit is reached.
• Interactive commands (e.g., python REPL, vim) are explicitly disabled; agents must rely on non-interactive file and search commands.
• Task-specific timeouts are enforced on training commands to prevent agents from bypassing evaluation by simply increasing model parameters.

Evidence (verbatim from paper)

A new evaluation metric inspired by AutoML and optimization literature enables fair cross-task comparison, while the framework supports flexible artifact evaluation (e.g., models, code, training logs) and algorithmic research (e.g., RL, curriculum learning).

Citation

@misc{nathani2025mlgym,
  title={MLGym: A New Framework and Benchmark for Advancing AI Research Agents},
  author={Nathani et al. (2025)},
  year={2025},
  note={arXiv:2502.14499}
}

• arXiv: 2502.14499