Trec2020 Deep Learning Eval

trec2020-deep-learning-eval

Overview of the TREC 2020 deep learning track — Craswell et al. (2021) (arXiv:2102.07662, 2021)

What this evaluates

Evaluates ad hoc information retrieval ranking methods on document and passage retrieval tasks using large-scale training data and human-labeled relevance judgments. It compares neural language models, neural networks, and traditional methods under blind single-shot conditions to assess ranking quality in the top-k results.

Datasets

• TREC 2020 Deep Learning Track — total ?; splits: test (-1)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computed using 4-level relevance judgments, discounting gains logarithmically by position to focus on the top results users see.
• AP — range: [0, 1]
  • Average Precision. Binarizes the 4-level relevance judgments into relevant/irrelevant to compute the area under the precision-recall curve.
• RR — range: [0, 1]
  • Reciprocal Rank. The inverse of the position of the first relevant document in the ranked list. Computed using both NIST and sparse MS MARCO labels.
• NCG@k — range: [0, 1]
  • Normalized Cumulative Gain at k (k=100 for documents, k=1000 for passages). Measures set-based quality without considering ranking order, using 4-level judgments.

Input / output format

Input: A query and a pre-provided list of top-k candidate documents (k=100) or passages (k=1000) to be ranked.

Output: A ranked list of the candidate documents or passages.

Scoring recipe

import math
def compute_ndcg_at_10(predictions, gold, k=10):
    dcg = 0.0
    for i, doc_id in enumerate(predictions[:k]):
        rel = gold.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    ideal_rels = sorted(gold.values(), reverse=True)[:k]
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• NCG@k is not supported in the standard trec_eval tool, so evaluators must implement it separately.
• RR is reported using two different relevance label sources (NIST vs sparse MS MARCO), which can yield different values.
• AP binarizes the 4-level relevance judgments, losing granularity compared to NDCG@10.

Evidence (verbatim from paper)

Our main metric in both tasks is Normalized Discounted Cumulative Gain (NDCG)—specifically, NDCG@10, since it makes use of our 4-level judgments and focuses on the first results that users will see. To get a picture of the ranking quality outside the top-10 we also report Average Precision (AP), although this binarizes the judgments. For comparison to the MS MARCO leaderboard, which often only has one relevant judgment per query, we report the Reciprocal Rank (RR) of the first relevant document on the NIST judgments, and also using the sparse leaderboard judgments.

Citation

@misc{craswell2021trec2020dl,
  title={Overview of the TREC 2020 deep learning track},
  author={Craswell et al. (2021)},
  year={2021},
  note={arXiv:2102.07662}
}

• arXiv: 2102.07662