Trec 2019 Dl Track Eval

trec-2019-dl-track-eval

Overview of the TREC 2019 deep learning track — Craswell et al. (2020) (arXiv:2003.07820, 2020)

What this evaluates

Evaluates ad-hoc information retrieval systems on document and passage ranking tasks using large-scale human-labeled judgments. It probes the ability of neural and traditional models to rank relevant items highly for a set of test queries.

Datasets

• TREC 2019 Deep Learning Track — total ?; splits: test (43)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computed over 4-level graded relevance judgments, discounting gains by log2(rank+1) and normalizing by the ideal DCG.
• NCG@100 — range: [0, 1]
  • Normalized Cumulative Gain at rank 100. Measures the cumulative gain of retrieved items discounted by rank, normalized by the ideal gain.
• AP — range: [0, 1]
  • Average Precision. Computes the mean of precision values at ranks where relevant documents appear.
• RR — range: [0, 1]
  • Reciprocal Rank. The inverse of the rank of the first relevant document in the list.

Input / output format

Input: Query string and a list of candidate documents or passages (top-k for reranking, or full candidate set for full-ranking).

Output: A ranked list of document or passage IDs corresponding to the input candidates.

Scoring recipe

def compute_ndcg_at_k(gold_relevance, pred_rank_order, k=10):
    ranked_golds = [gold_relevance[i] for i in pred_rank_order[:k]]
    dcg = sum(r / log2(i + 2) for i, r in enumerate(ranked_golds))
    ideal_golds = sorted(gold_relevance, reverse=True)[:k]
    idcg = sum(r / log2(i + 2) for i, r in enumerate(ideal_golds))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Confusing NIST labels (used for official track metrics) with MS MARCO labels (used for the RR (MS) column in results).
• Failing to distinguish between the 'rerank' subtask (re-ranking provided top-k candidates) and 'fullrank' subtask (generating candidates from scratch).
• Using binary relevance instead of the track's 4-level graded relevance judgments for NDCG computation.

Evidence (verbatim from paper)

Our main metric in both tasks is Normalized Discounted Cumulative Gain (NDCG)—specifically, NDCG@10, since it makes use of our 4-level judgments and focuses on the first results that users will see.

Citation

@misc{craswell2020overview,
  title={Overview of the TREC 2019 deep learning track},
  author={Craswell et al. (2020)},
  year={2020},
  note={arXiv:2003.07820}
}

• arXiv: 2003.07820