qhjqhj00 sciagent-scientific-reasoning-eval
SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning — Li et al. (2025) (arXiv:2511.08151, 2025)
What this evaluates
Evaluates a multi-agent LLM system's ability to solve high-level, cross-disciplinary scientific problems at Olympiad and frontier-exam levels. It probes formal reasoning, proof generation, symbolic derivation, and chemical modeling through adaptive routing and self-verification.
Datasets
- IMO 2025 — total ?; splits: test (-1)
- IMC 2025 — total ?; splits: test (-1)
- IPhO 2024 — total ?; splits: test (-1)
- IPhO 2025 — total ?; splits: test (-1)
- CPhO 2025 — total ?; splits: test (-1)
- IChO 2025 — total ?; splits: test (-1)
- HLE — total ?; splits: test (-1)
Metrics
Olympiad Scoring(primary) — range: 0-100 points- Sum of points awarded across all problem parts based on official competition scoring rubrics. Evaluated via an LLM grader trained on standard answers and scoring criteria, followed by human expert verification to ensure strict adherence to the rubrics.
HLE Consistency Score— range: other- Assesses consistency between model outputs and standard solutions for a subset of HLE mathematics and physics problems.
Input / output format
Input: Scientific problem statements from Olympiad competitions or HLE, including domain-specific constraints and required solution formats.
Output: Complete problem-solving process including intermediate reasoning steps, symbolic derivations, and final answer.
Scoring recipe
def score_olympiad(solution, rubric):
total_score = 0
for part in rubric.parts:
score = llm_grader.evaluate(solution, part.criteria)
total_score += score
verified_score = human_verify(total_score, rubric)
return verified_score
Common pitfalls
- AI graders may over-award points if not strictly constrained by the official rubric; human verification is required to prevent score inflation.
- Olympiad problems often have multiple valid solution paths; the evaluation must accept alternative correct derivations, not just match the standard answer verbatim.
- HLE subset selection is not fully specified, making exact replication of the general reasoning benchmark difficult.
Evidence (verbatim from paper)
Olympiad Scoring. For all Olympiad benchmarks, performance is measured using the official scoring criteria for each competition. Problems are typically divided into multiple parts (e.g., Part A, B, C), each with a specific point value. The total score for a problem $S_{\text{total}}$ is the sum of the points $S_{i}$ awarded for each part $i$.
Citation
@misc{li2025sciagent,
title={SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning},
author={Li et al. (2025)},
year={2025},
note={arXiv:2511.08151}
}
- arXiv: 2511.08151