qhjqhj00 mmteb-retrieval-eval
Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters — Tu et al. (2025) (arXiv:2510.14274, 2025)
What this evaluates
Evaluates the dense retrieval capability of multilingual embedding models across multiple languages and document-query pairs. It measures how well compact models can match or exceed larger baselines on standardized multilingual retrieval benchmarks.
Datasets
- MTEB (Multilingual) — total ?; splits: test (-1)
Metrics
MMTEB(Retrieval)(primary) — range: percent- Average retrieval score across the multilingual MTEB benchmark tasks. Typically computed as nDCG@10 per task and averaged across all languages/tasks.
Input / output format
Input: Query-document pairs for dense retrieval evaluation.
Output: Not specified in the provided text.
Scoring recipe
scores = []
for task in mmteb_retrieval_tasks:
scores.append(compute_ndcg_at_10(predictions, gold))
final_score = sum(scores) / len(scores)
return final_score
Common pitfalls
- Context length limitations of some baseline models (e.g., multilingual-e5-large-instruct supports ~500 tokens) can artificially cap retrieval performance.
- Data scale alone does not guarantee continuous gains; hard negative mining and task diversity are critical for further improvement.
Evidence (verbatim from paper)
| Model | #Params | Supported Context Length | Emb. Dim. | MMTEB(Retrieval) | | --- | --- | --- | --- | --- | | MGTE | 305M | 8k | 768 | 56.50 | | bge-m3 | 568M | 8k | 4096 | 55.59 | | multilingual-e5-large-instruct | 560M | 514 | 1024 | 57.12 | | snowflake-l | 568M | 8k | 1024 | 58.36 | | Our | 305M | 8k | 768 | 60.56 | | gte-Qwen2-1.5B-instruct | 1.5B | 32k | 1536 | 60.78 | | SFR-Embedding-Mistral | 7B | 4k | 4096 | 59.44 | | gte-Qwen2-7B-instruct | 7B | 32k | 3584 | 60.08
Table 1: Performance of models in the MTEB (Multilingual) retrieval tasks.
Citation
@misc{tu2025retrofitting,
title={Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters},
author={Tu et al. (2025)},
year={2025},
note={arXiv:2510.14274}
}
- arXiv: 2510.14274