Phopile Eval

phopile-eval

Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving — Shunfeng Zheng et al. (arXiv:2510.00919, 2025)

What this evaluates

This benchmark evaluates foundation models' ability to solve Olympiad-level physics problems using retrieval-augmented generation (RAG). It probes step-wise logical reasoning, correct application of physical laws, and robustness to noisy retrieved context. Performance is measured via an LLM-as-judge scoring framework that rewards both intermediate reasoning quality and final answer correctness.

Datasets

• PhoPile — total 390; splits: test (390); repo https://github.com/aialt/PhoPile

Metrics

• Average Score (AS) (primary) — range: percent
  • AS = (Total points gained by candidate model / (Number of questions × 10)) × 100%. Each problem is scored 0–10 by an LLM-as-judge (GPT-4) against a reference solution based on a standardized rubric.
• Pass Rate (PR) — range: percent
  • PR = 100% × (N_full-score / N_total). Counts the percentage of problems where the model achieves a perfect score of 10.

Input / output format

Input: Physics problem statement (text only or text with images/diagrams), optionally prepended with top-k retrieved question-answer pairs from a corpus.

Output: Step-wise reasoning chain followed by a final answer.

Scoring recipe

scores = []
for problem in dataset:
    score = llm_judge_score(problem.solution, problem.reference) # 0-10 scale
    scores.append(score)
N = len(scores)
AS = (sum(scores) / (N * 10)) * 100
PR = (sum(1 for s in scores if s == 10) / N) * 100

Common pitfalls

• Retrievers based on semantic similarity may return physically irrelevant problems, misleading the generator.
• Retrieved formats can cause models to output pedagogical guidance instead of direct final answers.
• Models may incorrectly treat conditions from retrieved problems as knowns for the target problem.

Evidence (verbatim from paper)

We report both the average score (AS) the LLMs and LMMs earn and their pass rate (PR) over PhoPile-Test and PhoPile(V)-Test. The average score is defined as, AS = (Total points gained by candidate model / (Number of questions * 10)) * 100%. We regard the generator (LLMs and LMMs) as successfully passing a problem if they answer the problem correctly and earn a score of 10. Therefore pass rate is defined as, PR = 100% * (N_full-score / N_total).

Citation

@misc{zheng2025phopile,
  title={Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving},
  author={Shunfeng Zheng et al.},
  year={2025},
  note={arXiv:2510.00919}
}

• arXiv: 2510.00919