Researchgym Eval

researchgym-eval

ResearchGym: Evaluating Language Model Agents on Real-World AI Research — Garikaparthi et al. (2026) (arXiv:2602.15112, 2026)

What this evaluates

Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints.

Datasets

• ResearchGym — total ?; splits: test (-1); repo https://github.com/Anikethh/ResearchGym

Metrics

• improvement over baselines (primary) — range: percent
  • Percentage improvement of the agent's final experimental result over the strong human baseline provided for each task. Calculated as ((agent_result - human_baseline) / human_baseline) * 100.
• sub-task completion rate — range: percent
  • Percentage of required research sub-tasks (e.g., hypothesis generation, experiment execution, analysis) successfully completed by the agent within the allocated budget and time.

Input / output format

Input: Real-world ML/AI research papers with methods withheld, accompanied by API access to HuggingFace, Semantic Scholar, Kaggle, and Exa search. Web search is filtered with an October 2024 cutoff and 160 paper-related URLs are blocked.

Output: Agent-generated hypotheses, executable experiment code/scripts, and final research submissions/outcomes.

Scoring recipe

def score(predictions, gold):
    # predictions: dict with 'final_result' and 'sub_task_status'
    # gold: dict with 'human_baseline' and 'sub_tasks' list
    improvement = ((predictions['final_result'] - gold['human_baseline']) / gold['human_baseline']) * 100
    completed = sum(1 for sub in gold['sub_tasks'] if predictions['sub_task_status'].get(sub, False))
    completion_rate = (completed / len(gold['sub_tasks'])) * 100
    return improvement, completion_rate

Common pitfalls

• Budget and time limits are strict and asymmetric: open agents get 10$ / 12hrs, while proprietary scaffolds get 20$ / 24hrs, heavily impacting long-horizon task completion.
• Web search is artificially constrained (Oct '24 cutoff, 160 blocked URLs), which may unfairly penalize agents relying on recent literature or specific repositories.
• Results are reported as best@k across 3 independent runs with mean ± std, not single-run accuracy, requiring careful aggregation and multiple trials.

Evidence (verbatim from paper)

GPT-5 achieves only 6.7% improvement over baselines across 15 runs and completes just 26.5% of sub-tasks, revealing persistent failure modes like impatience, poor resource management, overconfidence, and context limits—despite occasional state-of-the-art performance.

Citation

@misc{garikaparthi2026researchgym,
  title={ResearchGym: Evaluating Language Model Agents on Real-World AI Research},
  author={Garikaparthi et al. (2026)},
  year={2026},
  note={arXiv:2602.15112}
}

• arXiv: 2602.15112