Mme Unify Eval

mme-unify-eval

MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models — Xie et al. (2025) (arXiv:2504.03641, 2025)

What this evaluates

Evaluates unified multimodal large language models (U-MLLMs) on their ability to handle mixed-modality tasks that combine visual understanding, text generation, and sequential reasoning. It probes capabilities such as interleaved image-text generation, visual chain-of-thought reasoning, and image editing with explanations.

Datasets

• MME-Unify — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: percent
  • Standard accuracy calculated as the percentage of instances where the model's output exactly matches the ground truth answer or generated image.
• Acc+ — range: percent
  • Stricter accuracy metric that requires both the text reasoning component and the image generation component to be correct simultaneously.
• Unify Score — range: percent
  • Overall benchmark score aggregating performance across comprehension, generation, and unified task subdomains.

Input / output format

Input: Multimodal prompts consisting of images and text instructions/questions. Unified tasks may include reference images and complex, multi-step instructions requiring sequential reasoning.

Output: Text responses and/or generated images. Unified tasks require interleaved text and image outputs, or specific actions and coordinates.

Scoring recipe

def compute_metrics(predictions, golds):
    acc_list = []
    acc_plus_list = []
    for pred, gold in zip(predictions, golds):
        text_ok = (pred.text == gold.text)
        img_ok = (pred.image == gold.image)
        acc_list.append(1.0 if text_ok or img_ok else 0.0)
        acc_plus_list.append(1.0 if text_ok and img_ok else 0.0)
    return {
        'Acc': sum(acc_list) / len(acc_list) * 100,
        'Acc+': sum(acc_plus_list) / len(acc_plus_list) * 100
    }

Common pitfalls

• Evaluating only text or only image accuracy can mask severe deficiencies in the other modality, as models often excel at one while failing at the other.
• Multi-step Visual CoT tasks suffer from cascading errors; mistakes in early reasoning steps compound, causing final accuracy to drop to near zero even if intermediate outputs are partially correct.
• Complex instruction following for image generation (e.g., drawing auxiliary lines) is frequently ignored, with models producing style-biased or irrelevant images despite correct text reasoning.

Evidence (verbatim from paper)

For each unify task in Table[2], we require the models to generate the correct image and perform correct reasoning. Under these conditions, even for simple tasks such as answering common questions and generating images, the best open-sourced model (Anole) only achieves an accuracy of 59.65% and accuracy-plus of 38% (Table[3]).

Citation

@misc{xie2025mmeunify,
  title={MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2504.03641}
}

• arXiv: 2504.03641