qhjqhj00 neucir-2023-eval
Overview of the TREC 2023 NeuCLIR Track — Dawn Lawrie et al. (arXiv:2404.08071, 2024)
What this evaluates
Evaluates neural cross-language and multilingual information retrieval systems on news collections in Chinese, Persian, and Russian. It probes a model's ability to rank relevant documents when queries are in English and documents are in different languages, as well as its capacity to unify rankings across multiple languages.
Datasets
- NeuCLIR 2023 — total ?; splits: test (-1)
Metrics
nDCG@20(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain at rank 20. Computes the sum of graded relevance values of results up to position 20, discounted logarithmically by rank, and normalizes by the ideal DCG@20.
Input / output format
Input: English query (topic) and a document collection in Chinese, Persian, or Russian. For reranking, an initial ranked list of 1,000 documents is also provided.
Output: A ranked list of 1,000 document IDs. For the MLIR task, a single unified ranked list across all three language collections.
Scoring recipe
def compute_ndcg_at_20(relevance_scores, k=20):
dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
ideal = sorted(relevance_scores, reverse=True)[:k]
idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal))
return dcg / idcg if idcg > 0 else 0.0
Common pitfalls
- Ad-hoc CLIR requires searching the entire collection, while reranking CLIR only reorders a pre-provided list of 1,000 documents.
- MLIR requires producing a single unified ranked list across Chinese, Persian, and Russian collections, not separate per-language lists.
- Relevance judgments use a 4-point scale that is converted to a 3-point scale for qrels; failing to apply this conversion will skew metric values.
Evidence (verbatim from paper)
The CLIR runs summarized in Figure 1 substantially outperformed the monolingual runs this year; this is different from last year, when the top runs from CLIR and monolingual tasks had similar nDCG@20.
Citation
@misc{lawrie2024neucir,
title={Overview of the TREC 2023 NeuCLIR Track},
author={Dawn Lawrie et al.},
year={2024},
note={arXiv:2404.08071}
}
- arXiv: 2404.08071