Trec Dl Track Eval

trec-dl-track-eval

TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime — Craswell et al. (2021) (arXiv:2104.09399, 2021)

What this evaluates

Evaluates the reliability and best practices for using TREC Deep Learning test collections for ranking model evaluation. It probes whether researchers properly separate model selection from final evaluation and accounts for training variance.

Datasets

• TREC Deep Learning Track — total ?; splits: dev (-1), test (-1); repo https://github.com/bmitra-msft/TREC-Deep-Learning-Quick-Start

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It computes the weighted sum of graded relevance scores up to position 10, normalized by the ideal DCG@10 for the same query to yield a score between 0 and 1.

Input / output format

Input: Query, document collection, and pre-computed relevance judgments for each query-document pair.

Output: A ranked list of documents for each query.

Scoring recipe

def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum((2**rel - 1) / log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)
    idcg = sum((2**rel - 1) / log2(i + 2) for i, rel in enumerate(ideal[:k]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Selecting the best model checkpoint or architecture based on test set performance instead of a held-out dev set.
• Reporting dev set metrics as final results without a separate held-out test set.
• Relying on a single training run with one random seed, ignoring variance that can lead to false positive 'significant' differences.

Evidence (verbatim from paper)

The valid numbers to report are the TREC 2019 and TREC 2020 NDCG@10 results for ranker A, which can be read by following the dotted line from A to the two other curves.

Citation

@misc{craswell2021trecdl,
  title={TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime},
  author={Craswell et al. (2021)},
  year={2021},
  note={arXiv:2104.09399}
}

• arXiv: 2104.09399