qhjqhj00 ipho-2025-theory-eval
Physics Supernova: AI Agent Matches Elite Gold Medalists at IPhO 2025 — Qiu et al. (2025) (arXiv:2509.01659, 2025)
What this evaluates
Evaluates an AI agent's ability to solve complex, multi-part physics theory problems that require integrating visual data extraction, causal reasoning, and self-correction via tool use. The benchmark probes whether agentic architectures with domain-specific tools can match elite human performance on standardized physics competitions.
Datasets
- IPhO 2025 Theory Problems — total ?; splits: test (-1); repo https://github.com/CharlesQ9/Physics-Supernova
Metrics
score(primary) — range: [0, 30]- Total points awarded across all sub-questions based on an Olympiad-style rubric, normalized to a maximum of 30. Higher values indicate better alignment with expert solutions.
Input / output format
Input: A physics theory problem Q consisting of m sub-questions q_j, each accompanied by associated visual data s_j (e.g., figures, experimental results, schematic representations).
Output: Final answers for all sub-questions in Q, produced iteratively through a Reason-Act loop that incorporates tool observations and self-correction.
Scoring recipe
Initialize total_score = 0.
For each sub-question j in Q:
Evaluate agent's final answer against gold solution using Olympiad rubric.
If answer matches expected physical reasoning and value:
total_score += points_j
Return total_score (max 30).
Common pitfalls
- Visual data extraction errors from the ImageAnalyzer can cascade into incorrect reasoning if numeric values or graph trends are misread.
- Self-correction via the AnswerReviewer may fail to catch subtle physical inconsistencies, leading to unphysical results that still appear mathematically consistent.
- Scoring relies on Olympiad-style rubrics that may not be fully automated or explicitly detailed, making reproducibility dependent on human grading conventions.
Evidence (verbatim from paper)
enabling it to solve complex IPhO 2025 theory problems with gold-medalist-level performance (23.5/30, ranking 14th out of 406 humans), surpassing the median score of human gold medalists.
Citation
@misc{qiu2025physicssupernova,
title={Physics Supernova: AI Agent Matches Elite Gold Medalists at IPhO 2025},
author={Qiu et al. (2025)},
year={2025},
note={arXiv:2509.01659}
}
- arXiv: 2509.01659