Deepmath 103k Eval

deepmath-103k-eval

DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning — He et al. (2025) (arXiv:2504.11456, 2025)

What this evaluates

Evaluates mathematical reasoning capabilities on a curated, decontaminated dataset of challenging problems, measuring performance across standardized math competitions and academic benchmarks.

Datasets

• DeepMath-103K — total 103000; splits: train (-1); repo https://github.com/zwhe99/DeepMath

Metrics

• accuracy (primary) — range: percent
  • Percentage of problems where the model's final answer exactly matches the verified ground truth.

Input / output format

Input: Mathematical problem statements from DeepMath-103K, typically requiring step-by-step reasoning or direct answers.

Output: Model-generated solutions or final answers.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
return (correct / len(predictions)) * 100

Common pitfalls

• Benchmarks like AIME and AMC use exact-match or verified answer formats, not partial credit.
• Decontamination is critical; standard benchmarks often contain training data leakage, which this dataset explicitly addresses.

Evidence (verbatim from paper)

Table 5: Math reasoning performance after fine-tuning Qwen-2.5-7B via SFT. We also add DeepMath-Zero-7B as an RL counterpart for reference.

Citation

@misc{he2025deepmath103k,
  title={DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2504.11456}
}

• arXiv: 2504.11456