qhjqhj00 liveaopsbench-eval
Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation — Sadegh Mahdavi et al. (2025) (arXiv:2501.14275, 2025)
What this evaluates
Evaluates large language models' mathematical reasoning capabilities on Olympiad-level competition problems. It specifically probes whether models possess genuine problem-solving skills or merely rely on memorized pre-training data by using a continuously updated, timestamped benchmark to measure contamination-resistant accuracy.
Datasets
- AoPS24 — total ?; splits: test (-1)
- Math — total ?; splits: test (-1)
- OlympiadBench — total 675; splits: test (675)
- OmniMath — total 4428; splits: test (4428)
Metrics
accuracy(primary) — range: percent- Percentage of test instances where the model's final parsed answer exactly matches the ground truth answer. Calculated as (correct predictions / total predictions) * 100.
Input / output format
Input: Math problem text (question), formatted within the model's specific chat template (e.g., [INST] question [/INST]).
Output: Step-by-step solution and final answer, generated autoregressively within the model's chat template.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
pred_ans = extract_final_answer(pred)
gold_ans = extract_final_answer(gold)
if normalize(pred_ans) == normalize(gold_ans):
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Static benchmarks often suffer from pre-training contamination, inflating accuracy scores; this protocol mitigates it via timestamped data.
- Some Olympiad problems lack a single concrete final answer (e.g., proof-based questions), requiring manual 'no-answer' classification instead of strict exact-match.
- Benchmarks like OlympiadBench must be filtered to exclude questions containing images or figures to ensure text-only evaluation compatibility.
Evidence (verbatim from paper)
We also ablate the effect of solution rewriting, which is an important part of our pipeline. As shown by Figure [5(b)], rewriting solutions into a step-by-step format substantially improves the test accuracy across all benchmarks. ... verify the correctness of the final parsed answers by referring to the original post. As a result, we found that 92% of the annotations were marked as correct, while 5% were incorrect and 3% fell under the no-answer category.
Citation
@misc{mahdavi2025aops,
title={Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation},
author={Sadegh Mahdavi et al. (2025)},
year={2025},
note={arXiv:2501.14275}
}
- arXiv: 2501.14275