Neuclirbench Eval

neuclirbench-eval

NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval — Lawrie et al. (2025) (arXiv:2511.14758, 2025)

What this evaluates

Evaluates the ranking effectiveness of retrieval and reranking models across monolingual, cross-language, and multilingual information retrieval tasks. It specifically probes how well systems handle language mismatches and multilingual document collections without relying on simple keyword matching.

Datasets

• NeuCLIRBench — total 10000000; splits: test (-1)

Metrics

• nDCG@20 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20, measuring the quality of the ranked list against graded relevance judgments by discounting gains logarithmically with position.
• Judged@20 — range: [0, 1]
  • The proportion of the top-20 retrieved documents that have human relevance judgments available in the collection, indicating judgment coverage and reusability.

Input / output format

Input: Query text (in Chinese, Persian, Russian, or English) and a candidate document set (or full 10M-document corpus for first-stage retrieval).

Output: Ranked list of document IDs or relevance scores corresponding to the input candidate set.

Scoring recipe

def compute_ndcg_at_20(predicted_scores, relevance_labels, k=20):
    dcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(predicted_scores[:k]))
    ideal_labels = sorted(relevance_labels, reverse=True)[:k]
    idcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(ideal_labels))
    return dcg / idcg if idcg > 0 else 0.0

def compute_judged_at_20(top_k_docs, judgment_set):
    return sum(1 for doc in top_k_docs[:20] if doc in judgment_set) / 20.0

Common pitfalls

• Rerankers often fail to improve strong first-stage fusion baselines, as the initial ranking is already highly effective and difficult to beat.
• Multilingual retrieval tasks show significantly smaller performance gaps between models compared to monolingual or cross-language tasks, making differentiation harder.
• Directly comparing BM25 with query translation (QT) is unfair due to tripled query length, requiring careful baseline selection and normalization.

Evidence (verbatim from paper)

Models in each group are ordered by the nDCG@20 on the multilingual retrieval task.

Citation

@misc{lawrie2025neuclirbench,
  title={NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval},
  author={Lawrie et al. (2025)},
  year={2025},
  note={arXiv:2511.14758}
}

• arXiv: 2511.14758