Mm Bright Eval

mm-bright-eval

MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval — Abdallah et al. (2026) (arXiv:2601.09562, 2026)

What this evaluates

This benchmark evaluates reasoning-intensive retrieval capabilities across text-only and multimodal settings. It probes models' ability to align visual and textual information, navigate technical domain queries, and rank relevant documents or images based on complex, multi-modal prompts.

Datasets

• MM-BRIGHT — total 2803; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores for the top 10 retrieved items, normalized by the ideal DCG@10. Tasks 1-3 use binary relevance (0/1), while Task 4 uses graded relevance (0, 1, 2).

Input / output format

Input: A text query, optionally accompanied by one or more query images (for Tasks 2-4), and a candidate corpus of documents and/or images to retrieve from.

Output: A ranked list of retrieved corpus items (documents or images), evaluated at the top 10 positions.

Scoring recipe

def compute_ndcg_at_10(relevance_scores, top_k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:top_k]))
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores[:top_k]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Task 4 uses a graded relevance scale (rel=2,1,0) while Tasks 1-3 use binary labels, so scores across tasks are not directly comparable.
• Adding query images often degrades performance for current multimodal models compared to text-only baselines due to insufficient visual reasoning training.
• Models supporting only single-image inputs require vertical concatenation of query images, which can impact retrieval quality.

Evidence (verbatim from paper)

Following prior work (Thakur et al., 2021; Nguyen et al., 2016; Su et al., 2024), we use nDCG@10 as the primary metric. Tasks 1-3 use binary relevance labels following BEIR (Thakur et al., 2021). Task 4 uses graded relevance: rel=2 for gold passage with corresponding positive image, rel=1 for gold passage without image, and rel=0 for incorrect passages.

Citation

@misc{abdallah2026mmbright,
  title={MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval},
  author={Abdallah et al. (2026)},
  year={2026},
  note={arXiv:2601.09562}
}

• arXiv: 2601.09562