Mmlu Sr Eval

mmlu-sr-eval

MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models — Wang et al. (2024) (arXiv:2406.15468, 2024)

What this evaluates

This benchmark stress-tests the reasoning capability of large language models by replacing key terms in multiple-choice questions and answers with arbitrary dummy words and their definitions. It probes whether models rely on genuine conceptual understanding or merely on lexical memorization of pre-trained vocabulary. Performance is measured across three substitution variants to isolate the impact of context modification on reasoning robustness.

Datasets

• MMLU-SR — total ?; splits: Question Only (-1), Answer Only (-1), Question and Answer (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the proportion of correctly answered multiple-choice questions out of the total number of questions. Expressed as a decimal or percentage.

Input / output format

Input: Multiple-choice questions from MMLU where key terms in the question, answer choices, or both are replaced with arbitrary dummy words and their definitions. Models receive 5-shot examples from a development dataset.

Output: For GPT/Gemini: 'Answer: Letter of Choice.' For Llama3: Log-likelihood scores over answer choices to determine the selected letter.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return correct / len(predictions)

Common pitfalls

• Different inference strategies (generation vs. log-likelihood) are used across model families, which may not yield directly comparable accuracy scores.
• The 5-shot context uses examples from a development set, potentially introducing dataset-specific biases or leakage.
• Substituting terms in answer choices removes contextual aids present in questions, making the task inherently harder and potentially conflating reasoning ability with context-matching skills.

Evidence (verbatim from paper)

The data highlights how each model performs in the Humanities, Social Sciences, STEM, and Other academic fields, providing average scores for each subset. We observe consistent drop in model performance across all subsets when transitioning from the standard MMLU dataset to the more challenging MMLU-SR dataset, as evidenced by the decline in average accuracy from 0.771 on the MMLU dataset to 0.710, 0.655, and 0.585, on our MMLU-SR’s “Question Only”, “Answer Only”, and “Question and Answer” subsets, respectively, for the gpt-4o-mini model.

Citation

@misc{wang2024mmlusr,
  title={MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.15468}
}

• arXiv: 2406.15468