Olymmath Eval

olymmath-eval

Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models — Sun et al. (2025) (arXiv:2503.21380, 2025)

What this evaluates

Evaluates advanced mathematical reasoning capabilities on Olympiad-level problems. It probes a model's ability to perform rigorous, step-by-step logical deduction and numerical verification across algebra, geometry, number theory, and combinatorics. The benchmark also assesses cross-lingual reasoning performance between English and Chinese.

Datasets

• OlymMATH — total 400; splits: OlymMATH-EN-HARD (-1), OlymMATH-EN-EASY (-1), OlymMATH-ZH-HARD (-1), OlymMATH-ZH-EASY (-1); repo https://github.com/RUCAIBox/OlymMATH

Metrics

• Pass@1 (primary) — range: [0, 1]
  • Mean accuracy across all sampled responses for a given problem. Computed as the fraction of problems where the single sampled answer matches the gold answer.
• Cons@k — range: [0, 1]
  • Accuracy after applying majority voting across k sampled responses per problem. The consensus answer is the most frequent prediction, and accuracy is the fraction of problems where this consensus matches the gold answer.

Input / output format

Input: A mathematical problem statement presented in either English or Chinese.

Output: A final numerical answer, optionally preceded by a chain-of-thought reasoning trace.

Scoring recipe

def pass_at_1(predictions, gold):
    return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

def cons_at_k(predictions_list, gold, k):
    correct = 0
    for preds, g in zip(predictions_list, gold):
        consensus = max(set(preds), key=preds.count)
        if consensus == g: correct += 1
    return correct / len(gold)

Common pitfalls

• Models may use empirical guesses or heuristics that yield correct answers but lack rigorous reasoning, misleading rule-based or LLM-as-judge evaluators.
• Performance stability varies significantly with sample size; models evaluated with only 8 samples show higher variance than those with 64 samples.
• English-language benchmarks consistently yield higher scores due to pre-training data bias, which can skew cross-lingual comparisons if not accounted for.

Evidence (verbatim from paper)

For the Pass@1 metric, we compute the mean accuracy across all sampled responses to derive the final accuracy score. For the Cons@64 and Cons@8 metric, we implement majority voting to determine a consensus answer for each problem, subsequently calculating the average accuracy across the entire dataset.

Citation

@misc{sun2025olymmath,
  title={Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2503.21380}
}

• arXiv: 2503.21380