Math General Reasoning Eval

math-general-reasoning-eval

R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training — Li et al. (2026) (arXiv:2602.13103, 2026)

What this evaluates

Evaluates large language models on mathematical and general reasoning capabilities using a standardized suite of benchmarks. It measures the model's problem-solving accuracy under self-play training conditions, tracking sustained performance gains across multiple evolution iterations.

Datasets

• AMC — total ?; splits: test (-1)
• Minerva — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• Olympiad — total ?; splits: test (-1)
• AIME25 — total ?; splits: test (-1)
• AIME24 — total ?; splits: test (-1)
• SuperGPQA — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• BBEH — total ?; splits: test (-1)

Metrics

• pass@1 accuracy (primary) — range: [0, 1]
  • 1 if the single greedy-decoded model response exactly matches the ground truth answer, 0 otherwise. Averaged across instances.
• mean@32 — range: [0, 1]
  • Average accuracy computed over 32 independently sampled model responses per question. Used specifically for AMC and AIME benchmarks.
• Math AVG — range: [0, 1]
  • Arithmetic mean of pass@1 accuracy (or mean@32 for AIME/AMC) across the seven mathematical reasoning benchmarks.
• Overall AVG — range: [0, 1]
  • Arithmetic mean of pass@1 accuracy (or mean@32 for AIME/AMC) across all ten mathematical and general reasoning benchmarks.

Input / output format

Input: Natural language problem statements from mathematical and general reasoning benchmarks.

Output: Model-generated text response containing the final answer or solution steps.

Scoring recipe

def score_pass1(pred, gold):
    return 1.0 if normalize(pred) == normalize(gold) else 0.0

def score_mean32(preds, gold):
    correct = sum(1 for p in preds if normalize(p) == normalize(gold))
    return correct / 32.0

def compute_avg(accuracy_dict):
    return sum(accuracy_dict.values()) / len(accuracy_dict)

Common pitfalls

• Using greedy decoding for AMC and AIME instead of mean@32, which violates the specified evaluation protocol.
• Reporting unweighted averages across benchmarks with vastly different scales or question counts.
• Confusing pass@1 (single greedy sample) with pass@k or mean@k sampling strategies.

Evidence (verbatim from paper)

We evaluate on seven mathematical and three general reasoning benchmarks (App.[E]). We report pass@1 accuracy with greedy decoding for all benchmarks except AMC and AIME, where we use mean@32 following prior work*(Huang et al., [2025])*.

Citation

@misc{li2026rdiverse,
  title={R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.13103}
}

• arXiv: 2602.13103