Webfaq Retrieval Eval

webfaq-retrieval-eval

WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval — Dinzinger et al. (2025) (arXiv:2502.20936, 2025)

What this evaluates

Evaluates multilingual dense retrieval models on natural Q&A pairs by measuring ranking quality against gold answers. It tests the model's ability to retrieve relevant FAQ documents across multiple languages and assesses zero-shot generalization to other Wikipedia-based benchmarks.

Datasets

• WebFAQ — total ?; splits: test (-1), train (-1)
• Mr. TyDi — total ?; splits: test (-1)
• MIRACL (Hard Negatives) — total ?; splits: test (-1)

Metrics

• NDCG@10 (primary) — range: percent
  • Normalized Discounted Cumulative Gain at rank 10, calculated as the sum of graded relevance scores discounted by log2(rank+1), normalized by the ideal DCG. Reported as a percentage.

Input / output format

Input: Query string and a candidate document corpus (or document text) per language.

Output: Ranked list of top-10 documents or similarity scores for each query.

Scoring recipe

import math
def compute_ndcg_at_10(retrieved, relevant, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, d in enumerate(retrieved[:k]) if d in relevant)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant), k)))
    return (dcg / idcg) * 100.0 if idcg > 0 else 0.0

Common pitfalls

• Evaluating on Mr. TyDi and MIRACL is strictly zero-shot; models fine-tuned on WebFAQ may show different relative gains compared to in-domain WebFAQ test splits.
• BM25 baselines use fixed Pyserini defaults (k1=0.9, b=0.4); changing these will break reproducibility with the reported table.
• Hybrid retrieval combines dense cosine similarity and BM25 scores using lambda=1.1; missing scores for documents in only one set are set to zero.

Evidence (verbatim from paper)

Table[3] outlines retrieval performances on six languages – the intersection set of languages covered by WebFAQ, Mr. TyDi and MIRACL. Table[3]. Comparing retrieval performance on 3 multilingual datasets using NDCG@10 in %, including SotA embedding models and BM25 as baselines.

Citation

@misc{dinzinger2025webfaq,
  title={WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval},
  author={Dinzinger et al. (2025)},
  year={2025},
  note={arXiv:2502.20936}
}

• arXiv: 2502.20936