Naturalreasoning Eval

naturalreasoning-eval

NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions — Yuan et al. (2025) (arXiv:2502.13124, 2025)

What this evaluates

Evaluates the zero-shot reasoning capabilities of models trained via knowledge distillation or self-training on the NaturalReasoning dataset. It measures performance across diverse mathematics and science benchmarks to assess scaling efficiency and generalization.

Datasets

• MATH — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• GPQA-Diamond — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy: the proportion of instances where the model's predicted answer exactly matches the ground truth label.

Input / output format

Input: Zero-shot prompts for mathematics and science reasoning questions.

Output: Model-generated answers or reasoning traces.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Averaging accuracy over the last three saved checkpoints during training rather than the final model.
• Using greedy decoding for inference, which may penalize models that benefit from temperature sampling.
• Evaluating strictly in a zero-shot setting without few-shot examples or chain-of-thought prompting.

Evidence (verbatim from paper)

We evaluate our models on a diverse set of benchmarks that encompass both math and science reasoning: MATH, GPQA, GPQA-Diamond*(Rein et al., [2024])* and MMLU-Pro. To ensure a fair and consistent comparison, we adopt a zero-shot evaluation setting across all trained models. For inference we use vllm (Kwon et al., [2023]) and employ greedy decoding to maintain determinism and eliminate variability introduced by stochastic generation. Unless mentioned otherwise, we report accuracy averaged over the last three saved model checkpoints during training.

Citation

@misc{yuan2025naturalreasoning,
  title={NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2502.13124}
}

• arXiv: 2502.13124