Nemotron Math Eval

nemotron-math-eval

Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision — Du et al. (2025) (arXiv:2512.15489, 2025)

What this evaluates

Evaluates long-context mathematical reasoning and tool-integrated reasoning capabilities of language models on competition-style and open-domain advanced math problems. It probes symbolic precision, multi-step deduction, and the ability to leverage Python code execution for verification.

Datasets

• Comp-Math-24-25 — total ?; splits: test (-1)
• HLE-Math — total 976; splits: test (976)

Metrics

• pass@1 — range: [0, 1]
  • Measures the average accuracy across independent single-run generations per problem.
• maj@k (primary) — range: [0, 1]
  • Computes accuracy under majority voting over k generated solutions per problem. k=16 for AIME24/AIME25/HMMT-24-25 and k=4 for HLE-Math.

Input / output format

Input: Text-based mathematical problem statements (competition-style or open-domain), optionally with Python tool-integrated reasoning (TIR) enabled.

Output: Step-by-step reasoning trace and final answer. In TIR settings, includes executable Python code blocks.

Scoring recipe

def compute_metrics(predictions, gold_answers, k):
    maj_correct = []
    for preds, gold in zip(predictions, gold_answers):
        from collections import Counter
        vote = Counter(preds).most_common(1)[0][0]
        maj_correct.append(1 if vote == gold else 0)
    maj_at_k = sum(maj_correct) / len(maj_correct)
    pass_at_1 = sum(1 if p == gold else 0 for p, g in zip(predictions, gold_answers)) / len(predictions)
    return {'pass@1': pass_at_1, 'maj@k': maj_at_k}

Common pitfalls

• Using math-verify for HLE-Math instead of the specified LLM-as-a-judge protocol (Qwen2.5-32B-Instruct).
• Applying different decoding hyperparameters (temperature, top-p, max tokens) than the specified 1.0/1.0/120K, which invalidates maj@k comparisons.
• Confusing pass@1 (single-sample accuracy) with maj@k (majority voting over k samples), leading to incorrect reporting of model robustness.

Evidence (verbatim from paper)

We report both pass@1 and maj@k, where pass@1 measures the average accuracy across different independent runs and maj@k computes accuracy under majority voting over $k$ generated solutions (with $k=16$ for AIME24, AIME25, HMMT-24-25 and $k=4$ for HLE-Math).

Citation

@misc{du2025nemotronmath,
  title={Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2512.15489}
}

• arXiv: 2512.15489