M3retrieve Eval

m3retrieve-eval

M3Retrieve: Benchmarking Multimodal Retrieval for Medicine — Acharya et al. (2025) (arXiv:2510.06888, 2025)

What this evaluates

Evaluates the ability of multimodal retrieval models to accurately rank relevant medical documents in response to text-and-image queries. It probes domain-specific alignment, handling of complex clinical terminology, and cross-specialty generalization in safety-critical healthcare settings.

Datasets

• M3Retrieve — total 1200000; splits: test (-1); repo https://github.com/AkashGhosh/M3Retrieve

Metrics

• nNDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the predicted ranking by computing the DCG@10 of the retrieved documents and normalizing it by the ideal DCG@10 for the same set of relevant documents.

Input / output format

Input: Multimodal queries consisting of text and associated medical images, paired with a large corpus of text documents to be retrieved.

Output: A ranked list of retrieved document IDs or text snippets, typically evaluated at the top 10 positions.

Scoring recipe

def compute_nndcg_at_10(gold_relevant_set, predicted_ranking):
    dcg = 0.0
    for i, doc_id in enumerate(predicted_ranking[:10]):
        if doc_id in gold_relevant_set:
            dcg += 1.0 / math.log2(i + 2)
    ideal_dcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold_relevant_set), 10)))
    return dcg / ideal_dcg if ideal_dcg > 0 else 0.0

Common pitfalls

• Medical domain queries often contain complex or ambiguous terminology that can cause lexicon-based models like BM25 to fail without careful preprocessing or domain-specific tuning.
• Multimodal alignment issues arise when image-text relationships are weak or highly domain-specific, leading to poor cross-modal retrieval performance that standard CLIP-style models may not capture.
• nNDCG@10 assumes binary relevance; if the benchmark uses graded relevance scores, the formula must be adapted accordingly.

Evidence (verbatim from paper)

We used nNDCG@10 as the primary metric for evaluation. For both FLMR and BM25, the evaluation metrics were computed using the pytrec_eval Python library, following the implementation in the MTEB library.

Citation

@misc{acharya2025m3retrieve,
  title={M3Retrieve: Benchmarking Multimodal Retrieval for Medicine},
  author={Acharya et al. (2025)},
  year={2025},
  note={arXiv:2510.06888}
}

• arXiv: 2510.06888