Hipho Eval

hipho-eval

HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark? — Fangchen Yu et al. (2025) (arXiv:2509.07894, 2025)

What this evaluates

Evaluates multimodal large language models on authentic high school physics Olympiad problems, probing their ability to perform step-level physical reasoning, interpret complex diagrams and data plots, and solve problems across diverse physics subfields under Olympiad-level difficulty.

Datasets

• HiPhO — total ?; splits: test (-1); repo https://github.com/SciYu/HiPhO

Metrics

• Mean Normalized Score (MNS) (primary) — range: percent
  • MNS(M) = (1/N_M) * Σ_{Q∈M} (Exam Score(Q) / Full Mark(Q)) × 100%, where M is a category (modality type or physics field), N_M is the number of questions in M, and Q is a single question.
• Medal Threshold — range: other
  • Categorical ranking (Gold/Silver/Bronze) assigned based on total exam score relative to official Olympiad cutoffs. Used to compare model performance against human medalists.

Input / output format

Input: Physics Olympiad exam questions presented in mixed modalities: text-only, text with illustration figures, text with variable-based figures, or text with data figures. Experimental and diagram-generation problems are excluded to ensure fair evaluation.

Output: Step-by-step solutions and final numerical answers, graded against official Olympiad marking schemes.

Scoring recipe

def compute_mns(questions, category):
    scores = []
    for q in questions:
        if q.category == category:
            raw = grade_against_scheme(q.ground_truth, q.model_output)
            norm = (raw / q.full_mark) * 100
            scores.append(norm)
    return sum(scores) / len(scores) if scores else 0

def assign_medal(total_score):
    if total_score >= gold_cutoff: return 'Gold'
    elif total_score >= silver_cutoff: return 'Silver'
    else: return 'Bronze'

Common pitfalls

• Experimental and diagram-generation problems are explicitly excluded from evaluation, meaning the Full Mark for models is lower than for humans.
• Visual complexity significantly impacts scores; models drop sharply on variable-based and data figures compared to text-only or simple illustrations.
• Step-level grading uses official marking schemes rather than simple exact-match on final answers, requiring careful alignment with rubric points.

Evidence (verbatim from paper)

To investigate the impact of modality on performance, we categorize all problems into four types: Text-Only (TO), Text+Illustration Figure (TI), Text+Variable Figure (TV), and Text+Data Figure (TD). For each type, we report the Mean Normalized score (MNS), defined as: MNS(M) = (1/N_M)∑_{Q∈M} (Exam Score(Q)/Full Mark(Q))×100%, where M∈{TO,TI,TV,TD}, N_M is the number of questions in M, and Q denotes a single question.

Citation

@misc{yu2025hipho,
  title={HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?},
  author={Fangchen Yu et al. (2025)},
  year={2025},
  note={arXiv:2509.07894}
}

• arXiv: 2509.07894