# Mmdeepresearch Bench Eval

> Evaluates multimodal deep research agents on iterative retrieval, citation-grounded reasoning, and long-form report synthesis. It probes how well models align textual claims with visual evidence, maintain citation discipline, and produce high-quality structured reports under multimodal constraints. Use when the user wants to benchmark on MMDeepResearch-Bench, or asks about evaluating this task. Reports Overall MMDR-Bench Score.

- Skill: `qhjqhj00/mmdeepresearch-bench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mmdeepresearch-bench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mmdeepresearch-bench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mmdeepresearch-bench-eval

---


# mmdeepresearch-bench-eval

> MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents — Huang et al. (2026) (arXiv:2601.12346, 2026)

## What this evaluates

Evaluates multimodal deep research agents on iterative retrieval, citation-grounded reasoning, and long-form report synthesis. It probes how well models align textual claims with visual evidence, maintain citation discipline, and produce high-quality structured reports under multimodal constraints.

## Datasets

- **MMDeepResearch-Bench** — total 140; splits: test (140)

## Metrics

- `Overall MMDR-Bench Score` **(primary)** — range: [0, 100]
  - Weighted combination of three modules: 0.2 × FLAE + 0.5 × TRACE + 0.3 × MOSAIC. MOSAIC is gated with thresholds τ_F = τ_T = 0. Each module score is the mean of its sub-dimensions (e.g., FLAE = mean(Read, Insh, Stru)).
- `FLAE` — range: [0, 100]
  - Report quality module scoring Readability, Instruction Following, and Structure on a 0–100 scale.
- `TRACE` — range: [0, 100]
  - Citation alignment module scoring Visual Evidence Fidelity (Vef.), Consistency, Coverage, and Fidelity on a 0–100 scale.
- `MOSAIC` — range: [0, 100]
  - Text–visual consistency module scoring Semantic alignment, Accuracy, and VQA on a 0–100 scale.

## Input / output format

**Input**: Multimodal research task prompts containing textual queries and supporting visual evidence (images, charts, screenshots), often requiring iterative web search and retrieval.

**Output**: Long-form multimodal research report containing synthesized text, embedded figures/images, and citation-grounded claims with source URLs/snippets.

## Scoring recipe

```python
def compute_mmdr(flae, trace, mosaic):
    # Each module is the mean of its sub-dimensions (e.g., FLAE = mean(Read, Insh, Stru))
    # MOSAIC is gated with thresholds tau_F = tau_T = 0
    overall = 0.2 * flae + 0.5 * trace + 0.3 * max(mosaic, 0)
    return overall
# Unscorable cases receive a reason-aware validity penalty per Appendix A.5.2
```

## Common pitfalls

- Assuming multimodal inputs always improve performance; the benchmark shows vision can introduce noise and degrade citation discipline if visual grounding is unreliable.
- Equating high writing quality (FLAE) with faithful evidence use; strong prose does not guarantee accurate citation alignment or visual consistency (TRACE/MOSAIC).
- Overinterpreting absolute judge scores; different judge LLMs yield different absolute scores due to inductive biases, though relative rankings remain stable.

## Evidence (verbatim from paper)

> The overall MMDR-Bench score is a weighted combination of the three modules: FLAE (20%), TRACE (50%), and MOSAIC (30%). We assign the largest weight to TRACE because citation-grounded evidence quality is the most central requirement for deep research, while MOSAIC evaluates the additional report-quality constraints introduced by visual evidence.

## Citation

```bibtex
@misc{huang2026mmdeepresearchbench,
  title={MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2601.12346}
}
```

- arXiv: 2601.12346