Ir Metric Correlation Eval

ir-metric-correlation-eval

Correlation and Prediction of Evaluation Metrics in Information Retrieval — Kutlu et al. (2018) (arXiv:1802.00323, 2018)

What this evaluates

Evaluates the correlation and predictability of standard information retrieval metrics across multiple TREC test collections. It probes how well low-cost metrics can predict high-cost ones and how metric values vary between topic-wise and system-wise aggregations.

Datasets

• TREC Web & Robust Tracks (2000-2014) — total ?; splits: test (-1); repo https://github.com/trec-web/trec-web-2014

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision: average of precision values at ranks where relevant documents occur, averaged over topics.
• nDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain: compares the actual ranked list to an ideal one using logarithmic discounting at each rank.
• RBP@0.95 — range: [0, 1]
  • Rank-Biased Precision: models user browsing behavior with a persistence parameter p=0.95, weighting earlier ranks higher.
• ERR@20 — range: [0, 1]
  • Expected Reciprocal Rank at cutoff 20: estimates the probability that a user finds a relevant document at each position, discounted by position.
• P@10 — range: [0, 1]
  • Precision at rank 10: fraction of retrieved documents in the top 10 that are relevant.
• Recall@1000 — range: [0, 1]
  • Recall at cutoff 1000: fraction of all relevant documents retrieved within the top 1000 results.
• bpref — range: [0, 1]
  • Binary preference: proportion of pairs of relevant and non-relevant documents where the relevant one is ranked higher.
• RR — range: [0, 1]
  • Reciprocal Rank: inverse of the rank of the first relevant document.
• R-Prec — range: [0, 1]
  • Precision at the rank equal to the total number of relevant documents for the query.

Input / output format

Input: Ranked list of retrieved document IDs per query/topic, paired with ground-truth relevance judgments.

Output: Numerical score for each evaluation metric (e.g., MAP, nDCG, RBP) computed per system per topic or averaged across topics.

Scoring recipe

def compute_map(predictions, gold):
    hits = 0
    sum_prec = 0.0
    for i, doc in enumerate(predictions, 1):
        if doc in gold:
            hits += 1
            sum_prec += hits / i
    return sum_prec / len(gold) if gold else 0.0
# For other metrics (nDCG, RBP, ERR, etc.), use official trec_eval 9.0 or author-provided implementations with specified cutoffs and p values.

Common pitfalls

• Cut-off thresholds are metric-specific (e.g., ERR uses @20, while others default to @1000).
• RBP requires selecting a specific user behavior parameter p (0.5, 0.8, or 0.95).
• Runs with missing relevance judgments or identical ranked lists across collections must be filtered out to avoid bias.

Evidence (verbatim from paper)

Using the system runs submitted to these selected TREC tracks and their respective relevance judgments, we calculated 9 different evaluation metrics, including AP, bpref [24], ERR [25], nDCG, P@K, RBP [2], recall (R), RR [26], and R-Prec. We used various cut-off thresholds for the metrics... In our calculations, we test 0.5, 0.8 and 0.95 for the p parameter... We also detected runs that have identical ranked lists in WT2013 and WT2014 test collections and filtered out identical submissions.

Citation

@misc{kutlu2018correlation,
  title={Correlation and Prediction of Evaluation Metrics in Information Retrieval},
  author={Kutlu et al. (2018)},
  year={2018},
  note={arXiv:1802.00323}
}

• arXiv: 1802.00323