Hipho Physics Olympiad Eval

hipho-physics-olympiad-eval

PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System — Fangchen Yu et al. (2025) (arXiv:2509.24855, 2025)

What this evaluates

This benchmark evaluates multimodal physical reasoning and advanced problem-solving capabilities on international and regional physics Olympiad exams. It probes a model's ability to interpret complex diagrams, data, and text, perform multi-step logical derivations, and produce accurate solutions under strict, official scoring rubrics.

Datasets

• HiPhO — total ?; splits: test (-1)

Metrics

• exam score (primary) — range: other
  • Points awarded per exam based on official marking schemes, evaluated at both answer-level and step-level. Scores vary by exam maximum (e.g., 30 for IPhO, 25 for F=MA).
• Pass@k — range: other
  • The maximum exam score achieved across k independent inference attempts for a given problem or exam.

Input / output format

Input: Multimodal physics Olympiad problems containing text, illustrations, variable figures, and data figures.

Output: Step-by-step solutions and final answers to the physics problems.

Scoring recipe

def compute_exam_score(predictions, gold, marking_scheme):
    # Evaluate each prediction using official step-level and answer-level rubrics
    step_points = [score_steps(pred, gold, marking_scheme) for pred in predictions]
    answer_points = [score_final_answer(pred, gold) for pred in predictions]
    # Combine according to official marking scheme
    total_points = [s + a for s, a in zip(step_points, answer_points)]
    # Pass@k takes the highest score across k attempts
    return max(total_points)

Common pitfalls

• Evaluation temperature is fixed at 0.6, not sampled freely.
• Scoring uses both step-level and answer-level breakdowns per official marking schemes, not just exact-match final answers.
• Pass@k reports the best score across k attempts, not the average or median performance.

Evidence (verbatim from paper)

Pass@k evaluates a model’s best score over $k$ independent attempts by taking the highest-scoring solution per problem.

Citation

@misc{yu2025physicsminions,
  title={PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent System},
  author={Fangchen Yu et al. (2025)},
  year={2025},
  note={arXiv:2509.24855}
}

• arXiv: 2509.24855