Neuclir 2024 News Retrieval Eval

neuclir-2024-news-retrieval-eval

Overview of the TREC 2024 NeuCLIR Track — Dawn Lawrie et al. (arXiv:2509.14355, 2025)

What this evaluates

Evaluates neural cross-language and multilingual information retrieval systems on news collections in Chinese, Persian, and Russian. It probes a model's ability to rank documents by relevance when queries are in English and documents are in different languages, or when searching across multiple languages simultaneously.

Datasets

• NeuCLIR 1 News Collection — total ?; splits: test (-1)

Metrics

• nDCG@20 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20. Computed over graded relevance judgments mapped from a 4-point scale to a 3-point scale (3→3, 2→1, 1→0, 0→0). Divides the DCG@20 of the predicted ranking by the ideal DCG@20 based on the gold judgments.

Input / output format

Input: A topic (query) in English, and a document collection in Chinese, Persian, or Russian (for CLIR) or all three combined (for MLIR).

Output: A ranked list of up to 1,000 document IDs per topic, ordered by predicted relevance.

Scoring recipe

def compute_ndcg_at_20(predictions, gold_qrels):
    ndcg_scores = []
    for topic_id, ranked_docs in predictions.items():
        top_20 = ranked_docs[:20]
        dcg = sum(rel / math.log2(i + 2) for i, doc in enumerate(top_20) if (rel := gold_qrels[topic_id].get(doc, 0)) > 0)
        ideal_rels = sorted(gold_qrels[topic_id].values(), reverse=True)[:20]
        idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_rels))
        ndcg_scores.append(dcg / idcg if idcg > 0 else 0.0)
    return sum(ndcg_scores) / len(ndcg_scores)

Common pitfalls

• Relevance judgments use a converted 3-point scale (3→3, 2→1, 1→0, 0→0) from the original 4-point scale; failing to apply this mapping will skew nDCG scores.
• MLIR requires a single unified ranked list across all three language collections, not separate per-language lists.
• Topic filtering rules drop topics with >40% somewhat/very valuable documents or <2 relevant documents, which must be excluded from final metric computation.

Evidence (verbatim from paper)

For each topic, the system must return a ranked list of 1,000 documents drawn from the entire target language document collection, ordered by likelihood and degree of relevance to the topic. ... Figure 1. News CLIR nDCG@20. Coordinator runs are marked with slashes. Monolingual runs (i.e., using human-translated topics) are marked with “M” at the top of the bar.

Citation

@misc{lawrie2025neuclir,
  title={Overview of the TREC 2024 NeuCLIR Track},
  author={Dawn Lawrie et al.},
  year={2025},
  note={arXiv:2509.14355}
}

• arXiv: 2509.14355