Fire Bench Eval

fire-bench-eval

FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights — Wang et al. (arXiv:2602.02905, 2026)

What this evaluates

Evaluates autonomous coding agents on their ability to rediscover established scientific findings by autonomously planning, implementing, and executing experiments from scratch based only on high-level research questions. It probes end-to-end research workflow capabilities, including experimental design, code generation, and evidence-based conclusion formation.

Datasets

• FIRE-Bench — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • F1 score measuring rediscovery performance; exact calculation details are not specified in the provided section.

Input / output format

Input: Instruction file containing the research question, experimental constraints, and provided datasets, delivered to an agent in a sandbox environment via CLI.

Output: Autonomous execution trajectory including research planning, code implementation, experiment execution, and final evidence-based conclusions.

Scoring recipe

scores = []
for run in range(3):
    trajectory = agent.execute(task_instruction)
    score = compute_rediscovery_f1(trajectory, gold_finding)
    scores.append(score)
mean_score = sum(scores) / len(scores)
std_score = std(scores)
return mean_score, std_score

Common pitfalls

• Agents may bypass the rediscovery task by searching for the original paper online instead of generating an independent plan.
• Environment setup (e.g., package installation) is intentionally not preconfigured, testing the agent's full capability rather than just coding.
• Proprietary agent default checkpoints may change over time, affecting reproducibility across different evaluation windows.

Evidence (verbatim from paper)

Results show current agents achieve <50 F1 on rediscovery tasks, with dominant failure modes in research planning and conclusion formation, highlighting persistent gaps in scientific reasoning, experimental design, and evidence integration despite advances in LLM-powered agents.

Citation

@misc{wang2026firebench,
  title={FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights},
  author={Wang et al.},
  year={2026},
  note={arXiv:2602.02905}
}

• arXiv: 2602.02905