Physunibench Eval

physunibench-eval

PhysUniBench: An Undergraduate-Level Physics Reasoning Benchmark for Multimodal Models — Wang et al. (2025) (arXiv:2506.17667, 2025)

What this evaluates

Probes the ability of multimodal large language models to solve undergraduate-level physics problems that require integrating textual descriptions with complex diagrams. It specifically tests multi-step scientific reasoning, mathematical derivation, and conceptual understanding across eight distinct physics sub-disciplines.

Datasets

• PhysUniBench — total 3304; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances. For multiple-choice questions, it uses exact string matching against the correct option. For open-ended questions, it combines symbolic computation (e.g., SymPy) for mathematical equivalence with an LLM judge for reasoning and semantic correctness.

Input / output format

Input: Zero-shot setting: models receive a textual problem description paired with an associated image/diagram.

Output: For MC questions: a single selected option. For open-ended questions: the final answer enclosed in LaTeX \boxed{} format.

Scoring recipe

correct = 0
for pred, gold, q_type in zip(predictions, golds, types):
    if q_type == 'MC':
        if pred.strip() == gold.strip(): correct += 1
    else:
        pred_ans = extract_latex_box(pred)
        if sympy_equivalent(pred_ans, gold): correct += 1
        elif llm_judge_verifies(pred, gold): correct += 1
return (correct / len(predictions)) * 100

Common pitfalls

• Open-ended answers must strictly follow the \boxed{} LaTeX format; otherwise, automated parsing fails.
• The hybrid verification for open-ended questions (SymPy + LLM judge) introduces potential non-determinism and LLM bias not present in pure exact-match metrics.
• Evaluation is strictly zero-shot, so performance heavily depends on the model's inherent reasoning capabilities without in-context examples.

Evidence (verbatim from paper)

For MC questions, evaluation is based on exact matching with the correct answer. For OE questions, models must output their final answer using the LaTeX\boxed{} format. Answers are verified through symbolic computation (e.g., SymPy) for mathematical equivalence and a LLM judge (e.g., GPT-4o) for reasoning and semantic correctness. Model performance is reported in terms of accuracy, including overall accuracy across the entire benchmark, as well as accuracy broken down by physics sub-discipline, difficulty level, and question type (open-ended versus multiple-choice).

Citation

@misc{wang2025physunibench,
  title={PhysUniBench: An Undergraduate-Level Physics Reasoning Benchmark for Multimodal Models},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2506.17667}
}

• arXiv: 2506.17667