Usamo Proof Eval

usamo-proof-eval

Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad — Ivo Petrov et al. (2025) (arXiv:2503.21934, 2025)

What this evaluates

This benchmark probes the ability of large language models to generate rigorous, step-by-step mathematical proofs for high-school olympiad-level problems. It evaluates logical coherence, justification of assumptions, and adherence to formal proof standards rather than just numerical correctness.

Datasets

• 2025 USA Math Olympiad — total 6; splits: test (6); repo https://github.com/eth-sri/matharena

Metrics

• proof_points (primary) — range: [0, 42]
  • Sum of points awarded across all grading categories in the step-wise rubric. Maximum possible score is 42 points across the 6 problems.

Input / output format

Input: Problem statement embedded in a user prompt requesting a thorough, LaTeX-formatted answer with all proof steps included.

Output: LaTeX-formatted mathematical proof containing all logical steps, bounds, and justifications.

Scoring recipe

def compute_proof_points(solution, grading_scheme):
    total = 0
    for category in grading_scheme:
        awarded = judge_grade(solution, category['desc'])
        total += awarded
    return total

Common pitfalls

• Models often skip crucial logical steps or make unjustified assumptions, which heavily penalizes the score.
• Failing to rigorously prove bounds or choose sufficiently large N leads to lost points even if the final result is correct.
• LLM judges may struggle with subjective creativity requirements in proof construction, leading to inconsistent scoring.

Evidence (verbatim from paper)

The awarded number of points for each category should follow after the 'points' key - The reasoning for each decision should be explained after the 'desc' key.

Citation

@misc{petrov2025prooforbluff,
  title={Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad},
  author={Ivo Petrov et al. (2025)},
  year={2025},
  note={arXiv:2503.21934}
}

• arXiv: 2503.21934