Olympiad Math Eval

olympiad-math-eval

Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving — Songyang Gao et al. (arXiv:2512.10739, 2025)

What this evaluates

This evaluation probes an agent's ability to solve complex, long-horizon mathematical problems requiring multi-step deduction, proof construction, and lemma-based reasoning. It tests performance across standard competition benchmarks (AIME, HMMT) and rigorous Olympiad-level contests (IMO, CNMO, CMO), focusing on non-geometry problems where novel proof paths are required.

Datasets

• AIME2025 — total ?; splits: test (-1)
• HMMT2025 Feb — total ?; splits: test (-1)
• IMO2025 — total ?; splits: test (-1)
• CNMO2025 — total ?; splits: test (-1)
• CMO2025 — total 6; splits: test (6)

Metrics

• pass@1 (primary) — range: [0, 1]
  • Unbiased pass@k estimator: E[1/k * sum_{i=1}^k I(solution_i is correct)]. Correctness is determined by fine-grained grading points via CompassVerifier (solution-based) or OPV (proof-based). For IMO2025, pass@4 is used instead.

Input / output format

Input: Mathematical competition problems (solution-based and proof-based), excluding geometry for IMO2025 and CNMO2025.

Output: Final mathematical solution or proof, generated via multi-round hierarchical decomposition and lemma-based memory management.

Scoring recipe

def compute_passk(predictions, k, verifier_type):
    correct = []
    for pred in predictions[:k]:
        if verifier_type == 'solution':
            correct.append(1 if CompassVerifier(pred) else 0)
        else:
            correct.append(1 if OPV(pred) else 0)
    return sum(correct) / k
# Usage: k=1 for AIME/HMMT/CNMO/CMO; k=4 for IMO2025.
# For CMO2025, human experts grade per question (max 21 pts each, total 126).

Common pitfalls

• Geometry problems are explicitly excluded from IMO2025 and CNMO2025 evaluations.
• The pass@k value varies by dataset: pass@1 is used for AIME2025, HMMT2025, CNMO2025, and CMO2025, while pass@4 is strictly used for IMO2025.
• Two different verifiers are employed based on problem type: CompassVerifier for solution-based questions and OPV for proof-based questions.

Evidence (verbatim from paper)

For each sample, we perform 16 independent rollouts and use the unbiased pass@1 [chen2021evaluating] as the metric, except for IMO2025, which we use pass@4.

Citation

@misc{gao2025longhorizon,
  title={Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving},
  author={Songyang Gao et al.},
  year={2025},
  note={arXiv:2512.10739}
}

• arXiv: 2512.10739