qhjqhj00 hindi-beir-eval
Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5 — Acharya et al. (2024) (arXiv:2409.05401, 2024)
What this evaluates
Evaluates zero-shot information retrieval capabilities in Hindi across diverse domains and tasks. It probes how well multilingual embedding models and baselines rank relevant documents for Hindi queries without language-specific fine-tuning.
Datasets
- Hindi-BEIR — total ?; splits: test (-1); repo https://github.com/ArkadeepAcharya/NLLB-E5
Metrics
NDCG@10(primary) — range: [0, 1]- Normalised Discounted Cumulative Gain at rank 10. It measures ranking quality by summing the relevance score of each retrieved document discounted logarithmically by its position, then normalizing by the ideal DCG@10.
Input / output format
Input: Hindi query and a corresponding document collection (corpus) from one of the 15 Hindi-BEIR datasets.
Output: A ranked list of retrieved document IDs or text snippets for the given query.
Scoring recipe
def compute_ndcg_at_10(relevant_docs, retrieved_docs, k=10):
dcg = 0.0
for i, doc_id in enumerate(retrieved_docs[:k]):
rel = 1.0 if doc_id in relevant_docs else 0.0
dcg += rel / math.log2(i + 2)
idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
return dcg / idcg if idcg > 0 else 0.0
Common pitfalls
- Confusing Hindi-BEIR with the original English BEIR benchmark; Hindi-BEIR contains translated/curated Hindi versions of 15 specific IR tasks.
- Assuming models are fine-tuned on Hindi data; the benchmark specifically tests zero-shot performance on unseen Hindi domains.
- Using recall instead of NDCG@10; the paper explicitly states NDCG@10 is used because it is sensitive to ranking order.
Evidence (verbatim from paper)
Following BEIRThakur et al. ([2021]) benchmark standard, we use Normalised Discounted Cumulative Gain (NDCG)Järvelin and Kekäläinen ([2002]), more specifically NDCG@10 as our evaluation metric. NDCG is known to be a more robust metric than simple recall in the IR community because it is purposefully designed to be sensitive towards ranking of the retrieved results.
Citation
@misc{acharya2024hindi-beir,
title={Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5},
author={Acharya et al. (2024)},
year={2024},
note={arXiv:2409.05401}
}
- arXiv: 2409.05401