Imo2025 Eval

imo2025-eval

Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline — Huang et al. (2025) (arXiv:2507.15855, 2025)

What this evaluates

Evaluates a model's ability to solve Olympiad-level mathematical proof problems by generating rigorous solutions and iteratively refining them through self-verification.

Datasets

• IMO 2025 — total 6; splits: test (6); repo https://github.com/lyang36/IMO25

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of problems for which the pipeline accepts a solution. A solution is accepted only if it passes the verifier five consecutive times without critical errors or major justification gaps.

Input / output format

Input: Natural language statement of a mathematical proof problem from the IMO 2025 competition.

Output: A step-by-step mathematical proof or solution. The pipeline accepts the output if it robustly passes verification; otherwise it is rejected.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for problem, solution in zip(predictions, gold):
        passes = True
        for _ in range(5):
            if verifier(solution) != 'PASS':
                passes = False
                break
        if passes:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• The verifier is model-agnostic and runs iteratively; accuracy depends heavily on the number of verification passes (fixed at 5 in this work).
• Solutions are evaluated on mathematical rigor rather than just the final answer, so minor justification gaps may cause rejection even if the final result is correct.

Evidence (verbatim from paper)

We accept a solution if it robustly passes the verification process and decline a solution if there are always critical errors or major justification gaps during the iterations. ... we run the verifier five times and accept a solution only if it passes every time.

Citation

@misc{huang2025winninggoldimo2025,
  title={Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.15855}
}

• arXiv: 2507.15855