Trec Dl 2019 Eval

trec-dl-2019-eval

ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search — Craswell et al. (2020) (arXiv:2006.05324, 2020)

What this evaluates

Evaluates document ranking models on a small set of test queries from the TREC 2019 Deep Learning Track. It probes how effectively models trained on large-scale clicked query-document pairs can rerank documents according to human relevance judgments.

Datasets

• TREC 2019 Deep Learning Track — total 43; splits: test (43)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal of the rank of the first relevant document across all queries.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain: measures ranking quality by summing discounted relevance scores at each position, normalized by the ideal ranking.

Input / output format

Input: Query string and a candidate set of up to 100 documents (or document fields: URL, title, body, and optionally an ORCAS field).

Output: Relevance scores or a ranked list of the candidate documents.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    mrr_scores = []
    ndcg_scores = []
    for q in predictions:
        ranked_docs = q['predicted_rank']
        relevant_docs = q['gold_relevant_ids']
        for rank, doc_id in enumerate(ranked_docs, 1):
            if doc_id in relevant_docs:
                mrr_scores.append(1.0 / rank)
                break
        dcg = sum(1.0 / math.log2(rank + 1) for rank, doc_id in enumerate(ranked_docs[:10], 1) if doc_id in relevant_docs)
        idcg = sum(1.0 / math.log2(i + 1) for i in range(1, len(relevant_docs[:10]) + 1))
        ndcg_scores.append(dcg / idcg if idcg > 0 else 0.0)
    return sum(mrr_scores) / len(mrr_scores), sum(ndcg_scores) / len(ndcg_scores)

Common pitfalls

• The test set is extremely small (43 queries), making statistical significance difficult to establish.
• Negative sampling strategy during training (full collection vs. top-100) drastically changes baseline performance and must be explicitly matched when comparing models.
• ORCAS is used as training data or an auxiliary document field, not as a direct test benchmark.

Evidence (verbatim from paper)

Across both studies, we evaluate our models on the 43 test queries from the 2019 edition of the track using the corresponding NIST labels provided as a reusable benchmark. We report MRR and NDCG for each run.

Citation

@misc{craswell2020orcas,
  title={ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search},
  author={Craswell et al. (2020)},
  year={2020},
  note={arXiv:2006.05324}
}

• arXiv: 2006.05324