qhjqhj00 mldr-eval
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation — Chen et al. (2024) (arXiv:2402.03216, 2024)
What this evaluates
Evaluates retrieval over long multilingual documents (up to 8,192 tokens), testing a model's ability to capture information from extended contexts across multiple languages.
Datasets
- MLDR — total ?; splits: test (-1)
Metrics
nDCG@10(primary) — range: [0, 1]- Normalized Discounted Cumulative Gain at rank 10, measuring the quality of ranked long-document retrieval results.
Input / output format
Input: Long multilingual documents (from Wikipedia, Wudao, mC4) and corresponding queries.
Output: A ranked list of retrieved long documents.
Scoring recipe
def compute_ndcg_at_10(retrieved_ids, relevant_ids):
dcg = 0.0
for i, doc_id in enumerate(retrieved_ids[:10]):
if doc_id in relevant_ids:
dcg += 1.0 / math.log2(i + 2)
idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), 10)))
return dcg / idcg if idcg > 0 else 0.0
Common pitfalls
- Sparse retrieval surprisingly outperforms dense retrieval on this benchmark, contrary to typical dense-retrieval assumptions.
- The model's max length is fixed at 8192 tokens, which may truncate longer documents if not handled.
Evidence (verbatim from paper)
We evaluate the retrieval performance with longer sequences with two benchmarks: MLDR (Multilingual Long-Doc Retrieval), which is curated by the multilingual articles from Wikipedia, Wudao and mC4... measured by nDCG@10.
Citation
@misc{chen2024m3embedding,
title={M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
author={Chen et al. (2024)},
year={2024},
note={arXiv:2402.03216}
}
- arXiv: 2402.03216