Mmmu Pro Eval

mmmu-pro-eval

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark — Xiang Yue et al. (arXiv:2409.02813, 2024)

What this evaluates

This benchmark evaluates multimodal models' ability to perform robust, multi-discipline reasoning by forcing them to integrate visual and textual information without relying on shortcuts. It specifically probes resistance to guessing strategies through augmented multiple-choice options and tests true vision-text integration by embedding questions directly within images.

Datasets

• MMMU-Pro — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • The overall performance score is calculated as the average of accuracy scores from two settings: (2) Standard setting with augmented options (usually 10) and (3) Vision-only input setting where questions and options are embedded in images. Accuracy is computed as the proportion of correctly selected options.
• OCR Accuracy — range: percent
  • Computed as 1 - Levenshtein.distance(text1, text2) / max(len(text1), len(text2)), where text1 is the model-extracted text and text2 is the ground truth text from the image.

Input / output format

Input: Multimodal instances containing an image and a multiple-choice question. In the standard setting, the question and options are provided as text alongside the image. In the vision-only setting, both the question and all answer options are embedded directly within the image, with no explicit text input.

Output: The model must output a single selected option (e.g., A, B, C, D, etc.) corresponding to the correct answer. Models may be prompted with Direct or Chain-of-Thought (CoT) instructions, but the final evaluation uses the selected option.

Scoring recipe

def compute_mmmu_pro_score(preds_standard, gold_standard, preds_vision, gold_vision):
    acc_standard = sum(1 for p, g in zip(preds_standard, gold_standard) if p == g) / len(gold_standard)
    acc_vision = sum(1 for p, g in zip(preds_vision, gold_vision) if p == g) / len(gold_vision)
    overall_score = ((acc_standard + acc_vision) / 2) * 100
    return overall_score

Common pitfalls

• Confusing the standard 4-option setting with the benchmark's required 10-option augmented setting, which significantly changes the baseline guessing probability.
• Assuming high OCR accuracy correlates strongly with strong multimodal reasoning performance; the paper explicitly shows these metrics are largely independent.
• Using the original MMMU validation set as the primary benchmark instead of the combined MMMU-Pro score, which masks the increased difficulty and robustness requirements.

Evidence (verbatim from paper)

The overall performance score for MMMU-Pro is calculated as the average of scores from settings (2) and (3). We include setting (1) and report the original MMMU validation set performance solely for comparison purposes, to highlight the increased difficulty of MMMU-Pro.

Citation

@misc{yue2024mmmu_pro,
  title={MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark},
  author={Xiang Yue et al.},
  year={2024},
  note={arXiv:2409.02813}
}

• arXiv: 2409.02813