qhjqhj00 math-benchmarks-eval
OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data — Toshniwal et al. (2024) (arXiv:2410.01560, 2024)
What this evaluates
Evaluates the mathematical reasoning and problem-solving capabilities of language models across a spectrum of difficulties, from grade-school arithmetic to advanced competition-level mathematics.
Datasets
- GSM8K — total 1300; splits: test (-1)
- MATH — total 5000; splits: test (-1)
- AMC 2023 — total 40; splits: test (-1)
- AIME 2024 — total 30; splits: test (-1)
- Omni-MATH — total 4400; splits: test (-1)
Metrics
accuracy(primary) — range: [0, 1]- Proportion of correctly solved problems. Predicted answers are compared against ground truth using exact match or an LLM-as-a-judge (GPT-4o) to verify semantic equivalence.
Input / output format
Input: Natural language math problem statements.
Output: Step-by-step solutions and final answers, generated via greedy decoding or sampling (temperature 0.7, up to 256 samples).
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
pred_ans = extract_final_answer(pred)
if pred_ans == gold or llm_judge_gpt4o(pred_ans, gold):
correct += 1
return correct / len(predictions)
Common pitfalls
- Majority voting over 256 samples significantly increases inference cost and may mask single-pass greedy performance.
- Using GPT-4o as a judge for answer equivalence can introduce non-deterministic scoring compared to strict exact-match protocols.
- Zero-shot evaluation on competition datasets may be sensitive to prompt formatting variations not standardized in the paper.
Evidence (verbatim from paper)
We evaluate our models on a set of common benchmarks that consists of GSM8K (1.3K examples), MATH (5K examples), AMC 2023 (40 examples), AIME 2024 (30 examples), and Omni-MATH (4.4K examples) [26]. These datasets cover a broad spectrum of difficulty levels, ranging from grade school mathematics to advanced competition problems. Unless noted otherwise, all fine-tuned models are assessed in a zero-shot setting with both greedy decoding and majority voting out of 256 sampled solutions with temperature of 0.7 [32]. We use GPT-4o [27] as a judge to compare the ground truth answers with those predicted by our models (the detailed prompt is provided in Appendix D.5). ... the 8B model demonstrates strong accuracy gains compared to other LLMs of similar size
Citation
@misc{toshniwal2024openmathinstruct2,
title={OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data},
author={Toshniwal et al. (2024)},
year={2024},
note={arXiv:2410.01560}
}
- arXiv: 2410.01560