R Bench Eval

r-bench-eval

R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation — Guo et al. (2025) (arXiv:2505.02018, 2025)

What this evaluates

Evaluates complex reasoning capabilities of LLMs and MLLMs on graduate-level, multi-disciplinary academic questions in both English and Chinese. It probes the models' ability to handle rigorous, curriculum-based problems requiring extended chain-of-thought reasoning.

Datasets

• R-Bench-T — total 1094; splits: test (1094)
• R-Bench-M — total 665; splits: test (665)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Calculated as the percentage of questions where the model's predicted answer exactly matches the ground truth answer. Computed over the full test set and reported as a percentage.

Input / output format

Input: Text-only prompts for R-Bench-T; text-and-image prompts for R-Bench-M. Both include a Chain-of-Thought (CoT) instruction. Available in English and Chinese.

Output: Model generates a final answer (and optionally reasoning steps). For evaluation, the final answer is extracted and compared against the gold label.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Reasoning-focused models (e.g., o1-mini) do not benefit from explicit CoT prompting, unlike chat models, which can lead to misleading comparisons if prompting strategies are not standardized.
• Performance varies significantly across academic disciplines (up to 37.9% range), so aggregate scores may mask severe weaknesses in specific subjects.
• Multilingual consistency does not guarantee equivalent difficulty; models may overfit to specific linguistic patterns rather than demonstrating true cross-lingual reasoning transfer.

Evidence (verbatim from paper)

The values in the table represent the Top-1 accuracy, in %. In all tests, the CoT prompt is used by default. For details on the specific prompts, please refer to our appendix.

Citation

@misc{guo2025rbench,
  title={R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2505.02018}
}

• arXiv: 2505.02018