Zero Shot Retrieval Leakage Eval

zero-shot-retrieval-leakage-eval

How Train-Test Leakage Affects Zero-shot Retrieval — Fröbe et al. (2022) (arXiv:2206.14759, 2022)

What this evaluates

This evaluation probes the robustness of neural retrieval models to train-test data leakage by measuring how much performance on standard benchmarks artificially improves when training data contains near-duplicates or exact matches of test queries. It specifically assesses zero-shot transfer effectiveness under varying leakage conditions and training set sizes.

Datasets

• Robust04 — total ?; splits: test (172)
• TREC 2017 Common Core — total ?; splits: test (37)
• TREC 2018 Common Core — total ?; splits: test (38)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computed over the top-100 re-ranked documents for each query, comparing the model's predicted relevance scores against binary ground truth judgments.
• Precision@1 — range: [0, 1]
  • Binary metric that equals 1.0 if the top-ranked document is relevant, and 0.0 otherwise.
• MFR — range: other
  • Mean First Rank of a relevant document. Calculated as the average position of the first relevant document in the ranked list across all queries. Lower values indicate better performance, with 1.0 being optimal.

Input / output format

Input: Training: query paired with one relevant and one non-relevant document. Inference: query with a candidate list of top-100 BM25-ranked documents.

Output: A re-ranked list of the top-100 documents for each query, ordered by the model's predicted relevance score. Ties are broken via alphanumeric ordering by document ID.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of doc_ids ranked by model
    # gold: set of relevant doc_ids
    ndcg = calculate_ndcg_at_k(predictions, gold, k=10)
    prec1 = 1.0 if predictions[0] in gold else 0.0
    mfr = next(i+1 for i, doc in enumerate(predictions) if doc in gold)
    return ndcg, prec1, mfr
# Aggregate scores across queries, then compute mean across 5-fold CV splits.
# Significance tested via Student's t-test (p=0.05) with Bonferroni correction.

Common pitfalls

• MFR is inverted (lower is better), which can lead to misinterpretation if treated like standard accuracy metrics.
• Tie-breaking relies on alphanumeric document ID ordering rather than random or semantic similarity, which can artificially skew rankings for documents with similar scores.
• High nDCG@10 scores may reflect memorization of leaked queries rather than genuine zero-shot generalization capability.

Evidence (verbatim from paper)

We report the effectiveness of the models as nDCG@10, Precision@1, and the mean first rank of a relevant document (MFR) [18]. While effectiveness scores measured via nDCG@10 and Precision@1 have the property that higher values are better (a score of 1 indicates “best” effectiveness), for MFR, lower scores are better—but still a score of 1 is the best case indicating that the document on rank 1 always is relevant.

Citation

@misc{frobe2022leakage,
  title={How Train-Test Leakage Affects Zero-shot Retrieval},
  author={Fröbe et al. (2022)},
  year={2022},
  note={arXiv:2206.14759}
}

• arXiv: 2206.14759