Mmdocir Eval

mmdocir-eval

MMDocIR: Benchmarking Multimodal Retrieval for Long Documents — Dong et al. (2025) (arXiv:2501.08828, 2025)

What this evaluates

Evaluates multimodal retrieval systems on long documents by measuring their ability to retrieve relevant pages and fine-grained layout elements given a natural language query.

Datasets

• MMDocIR — total 73800; splits: train (73800), test (-1)

Metrics

• similarity scores (primary) — range: [0, 1]
  • Relevance of a query Q to a page p or layout l is computed as Sim(Q,p) or Sim(Q,l), used to rank items for top-k retrieval.

Input / output format

Input: Natural language query Q and a multimodal document corpus consisting of pages P and layout elements L.

Output: A ranked list of top-k pages and/or layouts most relevant to Q.

Scoring recipe

def score(predictions, gold, k):
    retrieved = predictions[:k]
    hits = sum(1 for pred in retrieved if pred in gold)
    return hits / len(gold)

Common pitfalls

• Converting multimodal documents to text via OCR or VLMs before retrieval causes information loss and reduces accuracy.
• Layout detection must correctly segment pages into 5-15 layout elements; errors here propagate to layout-level retrieval.

Evidence (verbatim from paper)

The relevance of pages (p) and layouts (l) to Q is measured by similarity scores, Sim(Q,p) and Sim(Q,l) respectively. The retrieval system consists of two phases: (1) an offline indexing phase, where pages and layouts from P and L are encoded into vectors, and (2) an online querying phase, in which a query Q is encoded into a vector, which is then compared against the offline-indexed vectors using similarity scores Sim(Q,p) for pages and Sim(Q,l) for layouts.

Citation

@misc{dong2025mmdocir,
  title={MMDocIR: Benchmarking Multimodal Retrieval for Long Documents},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2501.08828}
}

• arXiv: 2501.08828