Trec Dl Eval

trec-dl-eval

Conformer-Kernel with Query Term Independence for Document Retrieval — Mitra et al. (2020) (arXiv:2007.10434, 2020)

What this evaluates

Evaluates document retrieval models on a large-scale corpus by ranking millions of documents against a set of queries. It probes the model's ability to perform full-document retrieval and produce accurate ranked lists using explicit and latent matching signals.

Datasets

• TREC Deep Learning track (MS MARCO) — total 3213835; splits: train (384597), test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal ranks of the first relevant document across all queries. Calculated as 1/rank_of_first_relevant.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10: measures ranking quality by summing graded relevance scores discounted logarithmically by position, normalized by the ideal DCG@10.
• NCG@100 — range: [0, 1]
  • Normalized Cumulative Gain at rank 100: measures ranking quality by summing graded relevance scores up to rank 100, normalized by the ideal cumulative gain.

Input / output format

Input: Query (first 20 terms) and document text (first 4000 terms). Optional ORCAS click log field (max 2000 terms).

Output: Ranked list of documents from the full collection (3,213,835 documents) for each query.

Scoring recipe

def compute_metrics(predictions, gold):
    mrr, ndcg10, ncg100 = [], [], []
    for pred, rels in zip(predictions, gold):
        rr = 1.0 / (next(i for i, d in enumerate(pred) if d in rels) + 1)
        mrr.append(rr)
        dcg10 = sum(1.0 / (i + 1) for i, d in enumerate(pred[:10]) if d in rels)
        idcg10 = min(10, len(rels))
        dcg100 = sum(1.0 / (i + 1) for i, d in enumerate(pred[:100]) if d in rels)
        idcg100 = min(100, len(rels))
        ndcg10.append(dcg10 / idcg10 if idcg10 > 0 else 0)
        ncg100.append(dcg100 / idcg100 if idcg100 > 0 else 0)
    return {'MRR': sum(mrr)/len(mrr), 'NDCG@10': sum(ndcg10)/len(ndcg10), 'NCG@100': sum(ncg100)/len(ncg100)}

Common pitfalls

• Evaluating on a small candidate set instead of the full 3.2M document collection as specified.
• Using truncated document lengths (4000 terms) which may miss relevant content in long documents.
• Confusing NCG with NDCG; NCG does not apply position discounting.

Evidence (verbatim from paper)

We compare different runs based on following three metrics: mean reciprocal rank (MRR) [Craswell, 2009], normalized discounted cumulative gain (NDCG) [Järvelin and Kekäläinen, 2002], and normalized cumulative gain (NCG) [Rosset et al., 2018].

Citation

@misc{mitra2020conformerkernel,
  title={Conformer-Kernel with Query Term Independence for Document Retrieval},
  author={Mitra et al. (2020)},
  year={2020},
  note={arXiv:2007.10434}
}

• arXiv: 2007.10434