Ml Dev Bench Eval

ml-dev-bench-eval

ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows — Padigela et al. (2025) (arXiv:2502.00964, 2025)

What this evaluates

This benchmark evaluates AI agents' ability to execute end-to-end machine learning development workflows. It probes capabilities across six categories: dataset handling, model training, debugging, model implementation, API integration, and performance improvement. Success is measured by whether agents can produce fully functional, error-free code and configurations that satisfy the task specifications.

Datasets

• ML-Dev-Bench — total 30; splits: test (30); repo https://github.com/ml-dev-bench/ml-dev-bench

Metrics

• Success Rate (primary) — range: percent
  • Calculated as (Total Successful Tasks / Total Tasks) × 100%. Tasks are scored with a binary outcome: success (✓) if the agent completes the task accurately without errors or hallucinations, and failure (×) otherwise.

Input / output format

Input: Natural language task descriptions outlining ML development objectives (e.g., inspecting datasets, implementing training loops, debugging shape errors, modifying architectures, integrating APIs, or improving performance). Tasks often include specifications, code snippets, test cases, or dataset links, with difficulty levels varying by the amount of provided examples and hidden tests.

Output: Completed code files, scripts, or configuration changes that successfully execute the specified ML workflow. The output must be functionally correct, free of hallucinations, and pass the task's validation criteria.

Scoring recipe

def compute_success_rate(task_outcomes):
    # task_outcomes: list of booleans (True for success, False for failure)
    total = len(task_outcomes)
    if total == 0:
        return 0.0
    successful = sum(task_outcomes)
    return (successful / total) * 100

Common pitfalls

• Success is strictly binary; partial credit is not awarded, so minor bugs or incomplete implementations count as failure.
• Evaluation depends heavily on the execution environment and timeout settings, as tasks involve long-running training loops and multi-file debugging.
• Results vary significantly based on the agent-model pairing (e.g., ReAct vs. OpenHands with Sonnet vs. Gemini), making cross-agent comparisons sensitive to architectural choices rather than just model capability.

Evidence (verbatim from paper)

Tasks are evaluated based on binary success (✓) or failure (×). The aggregate success rate for each agent is calculated as: Success Rate = (Total Successful Tasks / Total Tasks) × 100%. Agents are assessed on their ability to complete tasks accurately without introducing errors or hallucinations.

Citation

@misc{padigela2025mldevbench,
  title={ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows},
  author={Padigela et al. (2025)},
  year={2025},
  note={arXiv:2502.00964}
}

• arXiv: 2502.00964