Mmmg Eval

mmmg-eval

MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning — Luo et al. (2025) (arXiv:2506.10963, 2025)

What this evaluates

This benchmark evaluates text-to-image reasoning capabilities by requiring models to generate domain-specific diagrams, charts, and mindmaps from vague prompts. It probes factual fidelity against annotated knowledge graphs and visual clarity across six educational tiers, revealing deficits in compositional planning and abstract reasoning.

Datasets

• MMMG — total 4456; splits: test (4456)

Metrics

• MMMG-Score (primary) — range: [0, 100]
  • A composite metric combining graph-edit distance (measuring factual accuracy against expert-annotated knowledge graphs) and SAM-2-based visual clarity metrics. Scores are scaled by 100 for reporting.

Input / output format

Input: A text prompt describing a domain-specific concept at a given educational tier, optionally accompanied by a reasoning trace (for reasoning-augmented models).

Output: A generated image (e.g., diagram, chart, mind map, or infographic) representing the prompt's content.

Scoring recipe

def compute_mmmg_score(generated_image, gold_kg):
    # Extract knowledge graph from generated image
    gen_kg = extract_knowledge_graph(generated_image)
    # Compute factual fidelity via graph-edit distance
    factual_score = 1.0 - (graph_edit_distance(gen_kg, gold_kg) / max_nodes)
    # Compute visual clarity using SAM-2 segmentation
    clarity_score = compute_sam2_clarity(generated_image)
    # Combine and scale to 0-100
    return (factual_score * 0.5 + clarity_score * 0.5) * 100

Common pitfalls

• LLM-as-a-judge (WIScore) and aesthetic metrics (AES-2.5) show poor correlation with human judgments for knowledge-dense visuals.
• FID is negatively correlated with human scores, making it an unreliable proxy for this task.
• Models may achieve high visual clarity but still fail on dependency structure/relationship accuracy, which the metric penalizes.

Evidence (verbatim from paper)

MMMG-Score combines graph-edit distance (for factual accuracy) and SAM-2-based visual clarity metrics, revealing severe reasoning deficits in state-of-the-art models (e.g., GPT-4o achieves only 50.20)... To assess alignment with human perception, we collected over 1,200 expert ratings (0–10 on clarity, correctness, accuracy and faithfulness) across six educational levels. We compared four metrics—MMMG-Score; an LLM-as-a-judge WIScore [27] with OpenAI-o3 evaluator; FID computed over 3,452 ground-truth images; and AES-2.5 [1]. Figure 8 reports their Pearson correlations against human scores: MMMG-Score leads with r = 0.876

Citation

@misc{luo2025mmmg,
  title={MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2506.10963}
}

• arXiv: 2506.10963