# Mmtr Bench Eval

> Evaluates Multimodal Large Language Models' ability to reconstruct masked text from visual context without explicit prompts. It probes layout understanding, visual grounding, and world knowledge integration by requiring models to infer missing content from surrounding text, charts, and multi-page evidence. Use when the user wants to benchmark on MMTR-Bench, or asks about evaluating this task. Reports exact-match / semantic-similarity.

- Skill: `qhjqhj00/mmtr-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mmtr-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mmtr-bench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mmtr-bench-eval

---


# mmtr-bench-eval

> Can MLLMs "Read" What is Missing? — Guo et al. (2026) (arXiv:2604.21277, 2026)

## What this evaluates

Evaluates Multimodal Large Language Models' ability to reconstruct masked text from visual context without explicit prompts. It probes layout understanding, visual grounding, and world knowledge integration by requiring models to infer missing content from surrounding text, charts, and multi-page evidence.

## Datasets

- **MMTR-Bench** — total 2771; splits: test (2771)

## Metrics

- `exact-match / semantic-similarity` **(primary)** — range: [0, 1]
  - Level-aware evaluation protocol: exact matching is applied to short targets (Level 1), while semantic similarity combined with a factuality gate is used for longer, complex reconstructions (Levels 2–4). An LLM-as-Judge performs automated assessment to determine the final score.

## Input / output format

**Input**: One or more images containing text with locally masked regions. No explicit question or prompt is provided.

**Output**: Recovered target text corresponding to the masked region(s).

## Scoring recipe

```python
def evaluate(predictions, gold, level):
    if level == 1:
        return 1.0 if predictions.strip() == gold.strip() else 0.0
    else:
        sim = compute_semantic_similarity(predictions, gold)
        factual = factuality_gate(predictions, gold)
        return sim if factual else 0.0
# Aggregate scores across all 2771 test samples; LLM-as-Judge used for final automated assessment
```

## Common pitfalls

- Assuming the task is standard VQA; there is no explicit question, requiring the model to autonomously identify relevant visual regions.
- Expecting random masking; masking is carefully curated by humans to ensure deterministic answers and clear contextual relationships.
- Overlooking the level-aware scoring; short targets use exact match while longer ones use semantic similarity + factuality, not a single uniform metric.

## Evidence (verbatim from paper)

> exact matching for short targets and semantic similarity with factuality gates for longer, complex reconstructions, revealing significant performance gaps between closed-source and open-source models, especially in sentence- and paragraph-level recovery.

## Citation

```bibtex
@misc{guo2026canmllmsread,
  title={Can MLLMs "Read" What is Missing?},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.21277}
}
```

- arXiv: 2604.21277