Mrmr Eval

mrmr-eval

MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval — Zhang et al. (2025) (arXiv:2510.09510, 2025)

What this evaluates

Evaluates reasoning-intensive multimodal retrieval across 23 expert domains using interleaved image-text queries and documents. It probes a model's ability to perform knowledge-based matching, theorem linking, and logical contradiction detection in complex, real-world scenarios.

Datasets

• MRMR — total 1502; splits: test (1502)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures the quality of the ranked list of retrieved documents by discounting relevance by position and normalizing by the ideal ranking.
• Hit@1 — range: [0, 1]
  • Binary metric indicating whether the single gold document appears at rank 1 in the retrieved list. Used exclusively for the Negation subtask.

Input / output format

Input: Interleaved image-text queries paired with multimodal documents. Evaluation setups vary: text retrievers use LLM-generated image captions, two-stream models use vector fusion of text and image embeddings, and document-as-image models encode entire documents as screenshots.

Output: A ranked list of retrieved documents or candidate documents for each query.

Scoring recipe

def compute_metrics(predictions, gold, task='default'):
    if task == 'negation':
        return 1.0 if predictions[0] == gold else 0.0
    k = 10
    dcg = sum(1.0 / math.log2(i + 2) for i, p in enumerate(predictions[:k]) if p == gold)
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The Negation subtask uses exactly 4 candidates per query, so Hit@1 scores below 25% indicate performance at or below random guessing.
• Models often rely on surface-level semantic matching rather than deep reasoning, leading to poor performance on Theorem and Contradiction subtasks despite strong results on Knowledge subtasks.
• Multimodal models struggle with expert-level query images (e.g., microscopic tissue sections) that require domain-specific reasoning beyond superficial visual similarity.

Evidence (verbatim from paper)

Following prior work (Xiao et al., 2025; Su et al., 2025), we use nDCG@10 as the main evaluation metric except Negation. Since each query in Negation has exactly one gold document among four candidates, we adopt Hit@1 as the main metric for this task.

Citation

@misc{zhang2025mrmr,
  title={MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.09510}
}

• arXiv: 2510.09510