LLM Reasoning Rl Eval

llm-reasoning-rl-eval

Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement — Lian (2025) (arXiv:2512.07611, 2025)

What this evaluates

This evaluation protocol assesses the mathematical reasoning and multi-step problem-solving capabilities of LLMs after reinforcement learning fine-tuning. It measures how well models generalize from a specialized arithmetic training task to standard academic and competitive math benchmarks.

Datasets

• GSM8K — total 8500; splits: test (-1)
• BBH — total ?; splits: test (-1)
• MATH — total 12500; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered problems out of the total number of problems in the benchmark split. Computed via exact match or normalized string comparison for open-ended math tasks, and option selection for multiple-choice tasks.

Input / output format

Input: Natural language math word problems, multi-step reasoning prompts, or multiple-choice questions presented as text.

Output: Text generation up to a maximum of 2048 tokens, containing the model's reasoning steps and final answer.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(predictions)) * 100

Common pitfalls

• Failing to cap the maximum output token length, which artificially inflates performance on math benchmarks (Table 2 shows gains up to 8192 tokens).
• Using inconsistent prompt formatting or stopping criteria, as LLM evaluation results are highly sensitive to these protocol choices.

Evidence (verbatim from paper)

Among the methods tested, DAPO without dynamic sampling emerged as the most effective algorithm, achieving the highest accuracy on all four benchmarks, including a peak of $53.3%$ on GSM8K and $30.0%$ on MMLU-Pro.

Citation

@misc{lian2025comparative,
  title={Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement},
  author={Lian (2025)},
  year={2025},
  note={arXiv:2512.07611}
}

• arXiv: 2512.07611