Reasoning Benchmarks Eval

reasoning-benchmarks-eval

Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training — Diao et al. (2025) (arXiv:2504.13161, 2025)

What this evaluates

Evaluates language model reasoning and general knowledge across multiple standard benchmarks. It measures the impact of data mixture optimization on model performance using accuracy scores on commonsense, scientific, and factual QA tasks.

Datasets

• PIQA — total ?; splits: test (-1)
• ARC_C — total ?; splits: test (-1)
• ARC_E — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• WinoGrande — total ?; splits: test (-1)
• SIQA — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• test accuracy (primary) — range: percent
  • Percentage of correctly answered questions on the test set. Calculated as (number of correct predictions / total number of examples) * 100.

Input / output format

Input: Multiple-choice questions or open-ended prompts formatted for the LM-Evaluation Harness, provided in a 0-shot setting (5-shot for MMLU).

Output: Model-generated text or selected answer option.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return (correct / len(gold_labels)) * 100

Common pitfalls

• Using validation splits for final reported results instead of test sets.
• Inconsistent shot settings across benchmarks (0-shot default vs 5-shot for MMLU) can skew comparisons if not explicitly noted.
• Averaging accuracy across benchmarks with different scales or question types without weighting.

Evidence (verbatim from paper)

For evaluation, we test on reasoning benchmarks: PIQA, ARC_C, ARC_E, HellaSwag, WinoGrande, and SIQA. We optimize using PIQA, ARC_E, and HellaSwag validation data, then evaluate on test sets. LM-Evaluation harness is used, with all datasets in a 0-shot setting except MMLU (5-shot).

Citation

@misc{diao2025nemotronclimb,
  title={Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training},
  author={Diao et al. (2025)},
  year={2025},
  note={arXiv:2504.13161}
}

• arXiv: 2504.13161