R2med Eval

r2med-eval

R2MED: A Benchmark for Reasoning-Driven Medical Retrieval — Li et al. (2025) (arXiv:2505.14558, 2025)

What this evaluates

Evaluates retrieval models on reasoning-driven medical tasks where document relevance is determined by alignment with inferred clinical diagnoses or multi-step reasoning paths rather than lexical or semantic overlap. Covers three task types—Q&A reference, clinical evidence, and clinical case retrieval—spanning eight medical sub-domains.

Datasets

• R2MED — total ?; splits: test (-1); repo https://github.com/R2MED/R2MED

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the ranked list of retrieved documents by discounting the relevance score logarithmically with position, normalized by the ideal DCG. Formula: nDCG@10 = DCG@10 / IDCG@10, where DCG@10 = Σ (rel_i / log2(i+1)) for i=1 to 10.

Input / output format

Input: A clinical query (question, case description, or evidence request) and a candidate corpus of medical documents.

Output: A ranked list of candidate documents (typically top-10 or top-100) ordered by predicted relevance to the query.

Scoring recipe

def compute_ndcg_at_10(retrieved_rel_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(retrieved_rel_scores[:k]))
    ideal_scores = sorted(retrieved_rel_scores, reverse=True)[:k]
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Assuming standard semantic retrieval models will perform well without explicit reasoning alignment, as lexical overlap is insufficient for clinical diagnosis inference.
• Reranking over top-100 candidates often degrades performance or yields inconsistent gains compared to reranking over top-10, especially for strong base retrievers.
• Medical-domain pretraining alone does not guarantee superior performance on reasoning-driven tasks if the training corpus lacks explicit reasoning paths.

Evidence (verbatim from paper)

Following prior work[nguyen2016ms] ; [thakur2021beir] ; [su2024bright], we use nDCG@10 as the primary evaluation metric.

Citation

@misc{li2025r2med,
  title={R2MED: A Benchmark for Reasoning-Driven Medical Retrieval},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.14558}
}

• arXiv: 2505.14558