Mrceval Eval

mrceval-eval

MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark — Ma et al. (2025) (arXiv:2503.07144, 2025)

What this evaluates

This benchmark evaluates large language models' ability to comprehend passages and answer questions across multiple dimensions, including context understanding, external knowledge integration, and complex reasoning. It probes factual fidelity, counterfactual handling, commonsense, world knowledge, and multi-hop reasoning capabilities.

Datasets

• MRCEval — total 2100; splits: test (-1); repo https://github.com/THU-KEG/MRCEval

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly answered questions out of the total number of questions.

Input / output format

Input: A reading comprehension passage, a question, and multiple-choice options, with a fixed instruction appended at the beginning.

Output: A single selected option corresponding to the correct answer.

Scoring recipe

correct = 0
total = len(gold)
for pred, gold_label in zip(predictions, gold):
    if pred.strip().lower() == gold_label.strip().lower():
        correct += 1
accuracy = (correct / total) * 100 if total > 0 else 0

Common pitfalls

• Models are evaluated with different sampling settings (DeepSeek uses temperature 0.6 and top-p 0.95, while all others use greedy/temperature 0.0), which may confound performance comparisons.
• Evaluation relies on a single run without averaging over multiple random seeds or temperature variations, making results sensitive to stochastic decoding.
• The fixed prompt is appended to every instance regardless of task type, which may artificially boost or suppress performance on certain sub-tasks.

Evidence (verbatim from paper)

We report accuracy as the metric from a single run result.

Citation

@misc{ma2025mrceval,
  title={MRCEval: A Comprehensive, Challenging and Accessible Machine Reading Comprehension Benchmark},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2503.07144}
}

• arXiv: 2503.07144