qhjqhj00 trec-2021-dl-eval
PASH at TREC 2021 Deep Learning Track: Generative Enhanced Model for Multi-stage Ranking — Qiao et al. (2022) (arXiv:2205.11245, 2022)
What this evaluates
Evaluates the ability of ranking models to retrieve and order relevant passages or documents for given queries. It probes multi-stage retrieval pipelines, including dense/sparse retrieval, query expansion, and re-ranking capabilities.
Datasets
- TREC 2021 Deep Learning Track — total ?; splits: test (-1)
Metrics
NDCG@5(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain at cutoff 5. Calculated as DCG@5 / IDCG@5, where DCG@5 = sum_{i=1}^{5} (2^{rel_i} - 1) / log2(i+1) and IDCG@5 is the DCG of the ideal ranking.
Input / output format
Input: Query text and a candidate set of passages or documents to be ranked.
Output: A ranked list of candidate passages or documents ordered by relevance score for each query.
Scoring recipe
def ndcg_at_k(ranked_rels, k=5):
dcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ranked_rels[:k]))
ideal_rels = sorted(ranked_rels, reverse=True)
idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels[:k]))
return dcg / idcg if idcg > 0 else 0.0
Common pitfalls
- Cutoff depth must be strictly enforced; scores beyond position k are ignored.
- Ties in retrieval scores must be broken consistently (e.g., randomly or by ID) as they directly impact ranking metrics like NDCG.
- Passage vs. document tasks require different aggregation levels; mixing them invalidates the evaluation.
Evidence (verbatim from paper)
achieving zero-shot NDCG@5 of 0.3651 for T5-11B. Ensemble learning across random seeds improves stability, with full-ranking runs achieving NDCG@5 of 0.7596 in passage and 0.7516 in document tasks.
Citation
@misc{qiao2022pash,
title={PASH at TREC 2021 Deep Learning Track: Generative Enhanced Model for Multi-stage Ranking},
author={Qiao et al. (2022)},
year={2022},
note={arXiv:2205.11245}
}
- arXiv: 2205.11245