Mmmu Eval

mmmu-eval — Multi-discipline multimodal expert-level QA

Source: Yue et al., "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI" (CVPR 2024, arXiv:2311.16502).

What this benchmark tests

College-level multimodal reasoning. 11,550 manually curated questions from textbooks, exams, quizzes across 6 disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Sci, Tech & Engineering), 30 subjects, 183 subfields. 30 image types — way beyond natural photos: chemical structures, geometric shapes, music sheets, MRI scans, circuit diagrams, technical schematics.

When to invoke this skill

• User wants to test an LMM on "expert-level" / "college-level" / "domain-knowledge" multimodal tasks
• User mentions MMMU, MMMU-Pro, MMMU-Plus, "expert AGI"
• User asks "how does my VLM stack up on technical disciplines?" (vs general visual QA like VQAv2)

Dataset structure

Each example has:

{
  "id": "Validation_1",
  "question": "Based on the chart, which company had the highest revenue in Q3?",
  "options": "['A) Acme', 'B) Globex', 'C) Initech', 'D) Umbrella']",
  "answer": "B",
  "question_type": "multiple-choice",        // or "open"
  "image_1": <PIL.Image>,                     // up to image_7 — interleaved
  "subject": "Finance",
  "subfield": "Corporate Finance",
  "img_type": "['Charts']",
  "topic_difficulty": "Medium"
}

Splits (HF MMMU/MMMU, 30 subject configs):

• dev: 150 (5/subject) — for prompt-tuning, public answers
• validation: 900 (30/subject) — public answers, report this for any model
• test: 10,500 — answers held out, must submit to leaderboard at https://mmmu-benchmark.github.io

Evaluation protocol (verbatim from §4)

1. Setting: zero-shot only; no fine-tuning on MMMU train data (there is none anyway).
2. Prompt: each model uses its own preferred prompt; for unfamiliar models, prompt-engineer on dev and reuse.
3. Question types:
   • Multi-choice (~94% of items): model outputs free text, scorer regexes for the option letter.
   • Open: model outputs free text, scorer regexes for key phrases / numbers.
4. Scoring (rule-based, not LLM-judge):
   • Multi-choice: extract first letter from response; if no valid letter, assign random answer (paper's tie-breaker).
   • Open: extract numeric / phrase via regex; if invalid, mark incorrect.
5. Metric: micro-averaged accuracy over all questions in the split.

Reference scoring (paper's official extractor, abridged)

import re
def extract_mc_answer(response: str, options: list[str]) -> str:
    # Try patterns "A)", "(A)", "Answer: A", final letter A/B/C/D
    for pat in [r"\b([A-Z])\)", r"\(([A-Z])\)", r"answer is\s*([A-Z])",
                r"\b([A-Z])\b\s*$"]:
        m = re.search(pat, response, re.IGNORECASE)
        if m and m.group(1).upper() in [chr(ord('A')+i) for i in range(len(options))]:
            return m.group(1).upper()
    import random; return random.choice([chr(ord('A')+i) for i in range(len(options))])

def micro_acc(predictions: list[dict], gold: list[dict]) -> float:
    correct = sum(p["pred"] == g["answer"] for p, g in zip(predictions, gold))
    return correct / len(gold)

Use the official scorer in the repo (mmmu/eval/eval_utils.py) when reporting numbers — it has many corner cases beyond the sketch above.

Output format the agent must produce

For each instance: {"id": "Validation_1", "response": "<free text>", "pred": "B"}.

When submitting to the test leaderboard, follow the JSON schema at https://github.com/MMMU-Benchmark/MMMU#leaderboard — micro-acc per discipline + overall.

Don'ts

• Don't fine-tune on dev/validation and report the same split.
• Don't use an LLM judge on this — the official protocol is rule-based; LLM-judge introduces noise that breaks comparability with leaderboard numbers.
• Don't ignore image_2..image_7 — many items are interleaved (e.g. multiple subfigures); concatenating images is wrong.
• Don't conflate MMMU and MMMU-Pro (which is a harder, deduplicated subset with different scores).

Citation

@inproceedings{yue2024mmmu,
  title={{MMMU}: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI},
  author={Yue, Xiang and Ni, Yuansheng and Zhang, Kai and others},
  booktitle={CVPR}, year={2024}
}