qhjqhj00 trec-covid-eval
TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection — Voorhees et al. (2020) (arXiv:2005.04474, 2020)
What this evaluates
Evaluates information retrieval systems on pandemic-related queries using a dynamically evolving corpus. It probes a system's ability to retrieve relevant medical literature under real-world conditions where terminology and document availability change rapidly.
Datasets
- TREC-COVID Round 1 — total ?; splits: test (30)
Metrics
NDCG@10(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list by comparing it to an ideal graded relevance ranking, discounting gains logarithmically by position and normalizing by the ideal DCG.
Input / output format
Input: A topic description (query) and a ranked list of candidate documents retrieved by the system.
Output: A ranked list of document IDs ordered by predicted relevance.
Scoring recipe
def compute_ndcg_at_10(relevance_scores, k=10):
dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
ideal = sorted(relevance_scores, reverse=True)
idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal[:k]))
return dcg / idcg if idcg > 0 else 0.0
# Map 'Partially Relevant' to 1, 'Relevant' to 2, unjudged/other to 0
Common pitfalls
- Relevance is graded ('Partially Relevant' vs 'Relevant'), not binary; treating all judged documents as binary relevant/non-relevant will distort NDCG scores.
- Depth-7 priority-based pooling means unjudged documents are not guaranteed to be non-relevant, violating the standard assumption required for unbiased estimation of some IR metrics.
Evidence (verbatim from paper)
Figure 5 provides a view of how effective the participants' systems were as a group. The figure contains a box-and-whiskers plot of the NDCG@10 scores across all 143 submitted runs for each topic.
Citation
@misc{voorhees2020trecovid,
title={TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection},
author={Voorhees et al. (2020)},
year={2020},
note={arXiv:2005.04474}
}
- arXiv: 2005.04474