qhjqhj00 trec2022-neuclir-eval
Overview of the TREC 2022 NeuCLIR Track — Dawn Lawrie et al. (2023) (arXiv:2304.12367, 2023)
What this evaluates
Evaluates neural cross-language information retrieval systems on ad hoc, reranking, and monolingual tasks across Chinese, Persian, and Russian newswire collections. It measures how well models retrieve and rank relevant documents when queries are in English and documents are in other languages, or when queries are human-translated.
Datasets
- TREC 2022 NeuCLIR Collections — total ?; splits: test (-1); repo https://github.com/NeuCLIR/download-collection
Metrics
nDCG@10(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain at rank 10. Standard TREC IR metric calculated as DCG@10 / IDCG@10, where DCG sums relevance scores discounted by log2(rank+1).
Input / output format
Input: English topics (or human-translated target-language topics for monolingual task) paired with document collections in Chinese, Persian, or Russian. For reranking, an initial BM25-ranked list of 1,000 documents is provided.
Output: A ranked list of 1,000 document IDs ordered by predicted relevance.
Scoring recipe
def compute_ndcg_at_10(predictions, gold):
dcg = 0.0
idcg = 0.0
for i, doc_id in enumerate(predictions[:10]):
rel = gold.get(doc_id, 0)
dcg += (2**rel - 1) / math.log2(i + 2)
sorted_rels = sorted(gold.values(), reverse=True)[:10]
for i, rel in enumerate(sorted_rels):
idcg += (2**rel - 1) / math.log2(i + 2)
return dcg / idcg if idcg > 0 else 0.0
Common pitfalls
- Reranking task uses a fixed initial run of 1,000 documents, not the full collection.
- Monolingual tasks require human-translated topics, not machine-translated ones.
- Runs using human-in-the-loop processes must be explicitly flagged as 'manual'.
Evidence (verbatim from paper)
Systems are then asked to rerank the documents to produce a new ordering that improves an evaluation metric. This task is suitable for teams that want to focus on second-stage scoring models, rather than on models which search an entire collection.
Citation
@misc{lawrie2023overview,
title={Overview of the TREC 2022 NeuCLIR Track},
author={Dawn Lawrie et al. (2023)},
year={2023},
note={arXiv:2304.12367}
}
- arXiv: 2304.12367