Math Reasoning Eval

math-reasoning-eval

Reasoning-Aware GRPO using Process Mining — Taekhyun Park, Yongjae Lee, Hyerim Bae (2025) (arXiv:2510.25065, 2025)

What this evaluates

Evaluates the mathematical reasoning capabilities of language models across multiple challenging benchmarks. It measures whether models can correctly solve math problems and follow structured reasoning processes aligned with a teacher model's trace.

Datasets

• MATH-500 — total ?; splits: test (-1)
• MINERVA — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)
• LiveMathBench — total ?; splits: test (-1)
• KSAT2025 — total ?; splits: test (-1)
• AIME 2024 — total ?; splits: test (-1)
• AIME 2025 — total ?; splits: test (-1)

Metrics

• Pass@1 (primary) — range: percent
  • Pass@1 measures the percentage of problems for which the first generated response exactly matches the ground truth answer after answer extraction.

Input / output format

Input: Mathematical problem statements or prompts requiring step-by-step reasoning and a final answer.

Output: A reasoning trace followed by a final extracted answer.

Scoring recipe

correct = 0
for problem, gold in zip(problems, gold_answers):
    response = model.generate(problem, temperature=0.6, top_p=0.95, max_tokens=16384)
    pred = extract_answer(response, library='DeepMath Evaluation Library')
    if pred == gold:
        correct += 1
pass_at_1 = (correct / len(problems)) * 100

Common pitfalls

• MATH-500 and AIME 2024 results are reported for reference only due to potential data contamination and should not be used for primary conclusions.
• Answer extraction must use the DeepMath Evaluation Library, as raw text matching will fail due to formatting variations in reasoning traces.

Evidence (verbatim from paper)

All evaluations used the Pass@1 metric temperature = 0.6, top_p = 0.95, and max_tokens = 16384, following the flashinfer*(Ye et al., [2025])* framework.\nThe answers are extracted using the DeepMath Evaluation Library*(He et al., [2025b])*.

Citation

@misc{park2025reasoningawaregrpo,
  title={Reasoning-Aware GRPO using Process Mining},
  author={Taekhyun Park, Yongjae Lee, Hyerim Bae (2025)},
  year={2025},
  note={arXiv:2510.25065}
}

• arXiv: 2510.25065