Mmmu Eval

mmmu-eval

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI — Xiang Yue et al. (2023) (arXiv:2311.16502, 2023)

What this evaluates

Evaluates expert-level multimodal understanding and reasoning across college-level disciplines. It probes a model's ability to interpret complex, domain-specific visual inputs combined with text, and apply specialized knowledge to solve multiple-choice or open-ended questions.

Datasets

• MMMU — total 11400; splits: validation (900), test (10500); repo https://mmmu-benchmark.github.io

Metrics

• micro-averaged accuracy (primary) — range: percent
  • Total correct predictions divided by total instances across all subjects and question types. Micro-averaging aggregates counts globally rather than averaging per-subject accuracies.

Input / output format

Input: Interleaved text and image inputs. Prompts follow the default format provided by each model for multi-choice or open QA tasks.

Output: Free-form text response containing reasoning steps, calculations, and a final answer or conclusion phrase.

Scoring recipe

def compute_accuracy(predictions, golds, types):
    correct = 0
    for pred, gold, qtype in zip(predictions, golds, types):
        ans = extract_regex(pred)
        if ans is None:
            ans = random.choice(options) if qtype == 'MC' else ''
        if match(ans, gold): correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models output long reasoning chains; evaluators must use robust regex pipelines to isolate the final answer, otherwise scoring fails.
• Open-ended questions have no fallback; if regex extraction fails, the response is strictly marked incorrect, unlike multiple-choice which defaults to random selection.
• Performance heavily depends on image type and discipline; reporting only overall accuracy masks severe weaknesses in specialized domains like Science or Health & Medicine.

Evidence (verbatim from paper)

We adopt micro-averaged accuracy as the evaluation metric. For both open and multiple-choice questions, we design systematic, rule-based evaluation pipelines. Specifically, to mitigate the potential influence of any intermediate generations (e.g., reasoning steps, calculations) in the long response, we construct robust regular expressions and develop response-processing workflows. These are employed to extract key phrases, such as numbers and conclusion phrases, from the long responses for accurate answer matching. If there is no valid answer in the model's response, we perform random selection as a remedy for multiple-choice questions or consider the response incorrect for open questions.

Citation

@misc{yue2023mmmu,
  title={MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
  author={Xiang Yue et al. (2023)},
  year={2023},
  note={arXiv:2311.16502}
}

• arXiv: 2311.16502