Mmtr Bench Eval

mmtr-bench-eval

Can MLLMs "Read" What is Missing? — Guo et al. (2026) (arXiv:2604.21277, 2026)

What this evaluates

Evaluates Multimodal Large Language Models' ability to reconstruct masked text from visual context without explicit prompts. It probes layout understanding, visual grounding, and world knowledge integration by requiring models to infer missing content from surrounding text, charts, and multi-page evidence.

Datasets

• MMTR-Bench — total 2771; splits: test (2771)

Metrics

• exact-match / semantic-similarity (primary) — range: [0, 1]
  • Level-aware evaluation protocol: exact matching is applied to short targets (Level 1), while semantic similarity combined with a factuality gate is used for longer, complex reconstructions (Levels 2–4). An LLM-as-Judge performs automated assessment to determine the final score.

Input / output format

Input: One or more images containing text with locally masked regions. No explicit question or prompt is provided.

Output: Recovered target text corresponding to the masked region(s).

Scoring recipe

def evaluate(predictions, gold, level):
    if level == 1:
        return 1.0 if predictions.strip() == gold.strip() else 0.0
    else:
        sim = compute_semantic_similarity(predictions, gold)
        factual = factuality_gate(predictions, gold)
        return sim if factual else 0.0
# Aggregate scores across all 2771 test samples; LLM-as-Judge used for final automated assessment

Common pitfalls

• Assuming the task is standard VQA; there is no explicit question, requiring the model to autonomously identify relevant visual regions.
• Expecting random masking; masking is carefully curated by humans to ensure deterministic answers and clear contextual relationships.
• Overlooking the level-aware scoring; short targets use exact match while longer ones use semantic similarity + factuality, not a single uniform metric.

Evidence (verbatim from paper)

exact matching for short targets and semantic similarity with factuality gates for longer, complex reconstructions, revealing significant performance gaps between closed-source and open-source models, especially in sentence- and paragraph-level recovery.

Citation

@misc{guo2026canmllmsread,
  title={Can MLLMs "Read" What is Missing?},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.21277}
}

• arXiv: 2604.21277