Mmdeepresearch Bench Eval

mmdeepresearch-bench-eval

MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents — Huang et al. (2026) (arXiv:2601.12346, 2026)

What this evaluates

Evaluates multimodal deep research agents on iterative retrieval, citation-grounded reasoning, and long-form report synthesis. It probes how well models align textual claims with visual evidence, maintain citation discipline, and produce high-quality structured reports under multimodal constraints.

Datasets

• MMDeepResearch-Bench — total 140; splits: test (140)

Metrics

• Overall MMDR-Bench Score (primary) — range: [0, 100]
  • Weighted combination of three modules: 0.2 × FLAE + 0.5 × TRACE + 0.3 × MOSAIC. MOSAIC is gated with thresholds τ_F = τ_T = 0. Each module score is the mean of its sub-dimensions (e.g., FLAE = mean(Read, Insh, Stru)).
• FLAE — range: [0, 100]
  • Report quality module scoring Readability, Instruction Following, and Structure on a 0–100 scale.
• TRACE — range: [0, 100]
  • Citation alignment module scoring Visual Evidence Fidelity (Vef.), Consistency, Coverage, and Fidelity on a 0–100 scale.
• MOSAIC — range: [0, 100]
  • Text–visual consistency module scoring Semantic alignment, Accuracy, and VQA on a 0–100 scale.

Input / output format

Input: Multimodal research task prompts containing textual queries and supporting visual evidence (images, charts, screenshots), often requiring iterative web search and retrieval.

Output: Long-form multimodal research report containing synthesized text, embedded figures/images, and citation-grounded claims with source URLs/snippets.

Scoring recipe

def compute_mmdr(flae, trace, mosaic):
    # Each module is the mean of its sub-dimensions (e.g., FLAE = mean(Read, Insh, Stru))
    # MOSAIC is gated with thresholds tau_F = tau_T = 0
    overall = 0.2 * flae + 0.5 * trace + 0.3 * max(mosaic, 0)
    return overall
# Unscorable cases receive a reason-aware validity penalty per Appendix A.5.2

Common pitfalls

• Assuming multimodal inputs always improve performance; the benchmark shows vision can introduce noise and degrade citation discipline if visual grounding is unreliable.
• Equating high writing quality (FLAE) with faithful evidence use; strong prose does not guarantee accurate citation alignment or visual consistency (TRACE/MOSAIC).
• Overinterpreting absolute judge scores; different judge LLMs yield different absolute scores due to inductive biases, though relative rankings remain stable.

Evidence (verbatim from paper)

The overall MMDR-Bench score is a weighted combination of the three modules: FLAE (20%), TRACE (50%), and MOSAIC (30%). We assign the largest weight to TRACE because citation-grounded evidence quality is the most central requirement for deep research, while MOSAIC evaluates the additional report-quality constraints introduced by visual evidence.

Citation

@misc{huang2026mmdeepresearchbench,
  title={MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2601.12346}
}

• arXiv: 2601.12346