Beir Nl Eval

beir-nl-eval

BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language — Banar et al. (2024) (arXiv:2412.08329, 2024)

What this evaluates

Evaluates zero-shot information retrieval capabilities of lexical, dense, and reranking models on Dutch-language queries and documents. It probes how well models generalize to a machine-translated benchmark without fine-tuning, measuring both ranking quality and recall performance.

Datasets

• MSMARCO — total ?; splits: test (-1)
• TREC-COVID — total ?; splits: test (-1)
• NFCorpus — total ?; splits: test (-1)
• NQ — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• FiQA-2018 — total ?; splits: test (-1)
• ArguAna — total ?; splits: test (-1)
• Touche-2020 — total ?; splits: test (-1)
• CQADupstack — total ?; splits: test (-1)
• Quora — total ?; splits: test (-1)
• DBPedia — total ?; splits: test (-1)
• SciDocs — total ?; splits: test (-1)
• SciFact — total ?; splits: test (-1)
• FEVER — total ?; splits: test (-1)
• Climate-FEVER — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures ranking quality by summing relevance gains discounted logarithmically by position, then normalizing by the ideal DCG. Higher values indicate better ranked results.
• Recall@100 — range: [0, 1]
  • The proportion of relevant documents retrieved within the top 100 results. It is ranking-agnostic and measures coverage of the relevant set.

Input / output format

Input: Dutch-language query and a corpus of Dutch documents.

Output: A ranked list of document IDs (or passages) for each query, evaluated at top-10 and top-100 positions.

Scoring recipe

def compute_metrics(predictions, gold_relevant_docs):
    top10 = predictions[:10]
    top100 = predictions[:100]
    # nDCG@10
    dcg10 = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(top10) if doc in gold_relevant_docs)
    idcg10 = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold_relevant_docs), 10)))
    ndcg10 = dcg10 / idcg10 if idcg10 > 0 else 0.0
    # Recall@100
    recall100 = len(set(top100) & set(gold_relevant_docs)) / len(gold_relevant_docs) if gold_relevant_docs else 0.0
    return ndcg10, recall100

Common pitfalls

• Models may show inflated performance if pre-trained or fine-tuned on data overlapping with the original English BEIR datasets, which are then translated into Dutch (contamination).
• Zero-shot evaluation strictly forbids fine-tuning on the target Dutch datasets; any in-domain tuning invalidates the zero-shot claim.
• Translation artifacts or machine translation errors in the benchmark can disproportionately affect lexical models like BM25 compared to dense models.

Evidence (verbatim from paper)

To assess the performance of our models, we employ two standard retrieval metrics: nDCG@10 and Recall@100. NDCG (normalized discounted cumulative gain) is a ranking-aware metric often used to report retrieval performance, especially on graded (non-binary) labels Thakur et al. ([2021]). We also report recall, which, although ranking-agnostic, is a useful and relevant metric for practical settings like retrieval-augmented generation.

Citation

@misc{banar2024beirnl,
  title={BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language},
  author={Banar et al. (2024)},
  year={2024},
  note={arXiv:2412.08329}
}

• arXiv: 2412.08329