# Deepmath 103k Eval

> Evaluates mathematical reasoning capabilities on a curated, decontaminated dataset of challenging problems, measuring performance across standardized math competitions and academic benchmarks. Use when the user wants to benchmark on DeepMath-103K, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/deepmath-103k-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/deepmath-103k-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/deepmath-103k-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/deepmath-103k-eval

---


# deepmath-103k-eval

> DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning — He et al. (2025) (arXiv:2504.11456, 2025)

## What this evaluates

Evaluates mathematical reasoning capabilities on a curated, decontaminated dataset of challenging problems, measuring performance across standardized math competitions and academic benchmarks.

## Datasets

- **DeepMath-103K** — total 103000; splits: train (-1); repo https://github.com/zwhe99/DeepMath

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of problems where the model's final answer exactly matches the verified ground truth.

## Input / output format

**Input**: Mathematical problem statements from DeepMath-103K, typically requiring step-by-step reasoning or direct answers.

**Output**: Model-generated solutions or final answers.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
return (correct / len(predictions)) * 100
```

## Common pitfalls

- Benchmarks like AIME and AMC use exact-match or verified answer formats, not partial credit.
- Decontamination is critical; standard benchmarks often contain training data leakage, which this dataset explicitly addresses.

## Evidence (verbatim from paper)

> Table 5: Math reasoning performance after fine-tuning Qwen-2.5-7B via SFT. We also add DeepMath-Zero-7B as an RL counterpart for reference.

## Citation

```bibtex
@misc{he2025deepmath103k,
  title={DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2504.11456}
}
```

- arXiv: 2504.11456