# Researchgym Eval

> Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints. Use when the user wants to benchmark on ResearchGym, or asks about evaluating this task. Reports improvement over baselines.

- Skill: `qhjqhj00/researchgym-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/researchgym-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/researchgym-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/researchgym-eval

---


# researchgym-eval

> ResearchGym: Evaluating Language Model Agents on Real-World AI Research — Garikaparthi et al. (2026) (arXiv:2602.15112, 2026)

## What this evaluates

Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints.

## Datasets

- **ResearchGym** — total ?; splits: test (-1); repo https://github.com/Anikethh/ResearchGym

## Metrics

- `improvement over baselines` **(primary)** — range: percent
  - Percentage improvement of the agent's final experimental result over the strong human baseline provided for each task. Calculated as ((agent_result - human_baseline) / human_baseline) * 100.
- `sub-task completion rate` — range: percent
  - Percentage of required research sub-tasks (e.g., hypothesis generation, experiment execution, analysis) successfully completed by the agent within the allocated budget and time.

## Input / output format

**Input**: Real-world ML/AI research papers with methods withheld, accompanied by API access to HuggingFace, Semantic Scholar, Kaggle, and Exa search. Web search is filtered with an October 2024 cutoff and 160 paper-related URLs are blocked.

**Output**: Agent-generated hypotheses, executable experiment code/scripts, and final research submissions/outcomes.

## Scoring recipe

```python
def score(predictions, gold):
    # predictions: dict with 'final_result' and 'sub_task_status'
    # gold: dict with 'human_baseline' and 'sub_tasks' list
    improvement = ((predictions['final_result'] - gold['human_baseline']) / gold['human_baseline']) * 100
    completed = sum(1 for sub in gold['sub_tasks'] if predictions['sub_task_status'].get(sub, False))
    completion_rate = (completed / len(gold['sub_tasks'])) * 100
    return improvement, completion_rate
```

## Common pitfalls

- Budget and time limits are strict and asymmetric: open agents get 10$ / 12hrs, while proprietary scaffolds get 20$ / 24hrs, heavily impacting long-horizon task completion.
- Web search is artificially constrained (Oct '24 cutoff, 160 blocked URLs), which may unfairly penalize agents relying on recent literature or specific repositories.
- Results are reported as best@k across 3 independent runs with mean ± std, not single-run accuracy, requiring careful aggregation and multiple trials.

## Evidence (verbatim from paper)

> GPT-5 achieves only 6.7% improvement over baselines across 15 runs and completes just 26.5% of sub-tasks, revealing persistent failure modes like impatience, poor resource management, overconfidence, and context limits—despite occasional state-of-the-art performance.

## Citation

```bibtex
@misc{garikaparthi2026researchgym,
  title={ResearchGym: Evaluating Language Model Agents on Real-World AI Research},
  author={Garikaparthi et al. (2026)},
  year={2026},
  note={arXiv:2602.15112}
}
```

- arXiv: 2602.15112