qhjqhj00 ms-marco-v2-ranking-eval
Overview of the TREC 2023 deep learning track — Craswell et al. (2025) (arXiv:2507.08890, 2025)
What this evaluates
Evaluates passage and document ranking systems on a large-scale, document-native corpus. It probes a model's ability to retrieve relevant content from millions of documents using sparse, crowd-sourced relevance judgments, while handling realistic corpus drift and query-independent passage extraction.
Datasets
- MS MARCO v2 — total 11900000; splits: train (-1), dev (-1), eval (-1)
Metrics
NDCG@10(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list by comparing the discounted cumulative gain of the predicted ranking against the ideal ranking derived from relevance judgments.
Input / output format
Input: A natural language query and a candidate set of passages or documents (up to 1000 for reranking, or full corpus for full ranking).
Output: A ranked list of passages or documents ordered by predicted relevance to the query.
Scoring recipe
def compute_ndcg_at_10(pred_ranking, gold_relevance, k=10):
dcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(pred_ranking[:k]))
ideal = sorted(gold_relevance, reverse=True)[:k]
idcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(ideal))
return dcg / idcg if idcg > 0 else 0.0
Common pitfalls
- Relevance judgments (qrels) are highly sparse and derived from crowd workers selecting from only 10 initial results, not exhaustive ground truth.
- Passage-to-document mapping assumes relevance transfers if passages share a URL and sufficient text similarity, which may introduce noise.
- Corpus content drift: documents extracted later may have changed or disappeared, requiring systems to handle stale or missing content.
Evidence (verbatim from paper)
We can consider how big this problem is by analyzing the disagreement rate between MS MARCO qrels and NIST qrels (in v1 and v2), and seeing whether training on MS MARCO qrels yields improved NIST NDCG on the test set.
Citation
@misc{craswell2025trec2023dl,
title={Overview of the TREC 2023 deep learning track},
author={Craswell et al. (2025)},
year={2025},
note={arXiv:2507.08890}
}
- arXiv: 2507.08890