Mmteb Multilingual Eval

mmteb-multilingual-eval

Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks — Babakhin et al. (2025) (arXiv:2511.07025, 2025)

What this evaluates

Evaluates multilingual and cross-lingual text embedding capabilities across 131 tasks spanning 250+ languages. Probes performance on diverse NLP tasks including retrieval, classification, clustering, and semantic textual similarity using instruction-tuned embeddings.

Datasets

• MTEB Multilingual (MMTEB) — total ?; splits: test (-1)

Metrics

• Borda count (primary) — range: other
  • Each of the 131 tasks acts as a voter, assigning points to models based on their relative performance rank on that task. The model with the highest total points across all tasks achieves the top rank, rewarding broad and consistent generalization.
• Mean(Task) — range: percent
  • The arithmetic average of the performance scores across all 131 individual tasks.
• Mean(Type) — range: percent
  • The arithmetic average of the performance scores across the 9 problem types.

Input / output format

Input: Text pairs (query/document) accompanied by task-specific instructions or templates.

Output: Fixed-dimensional embedding vectors for each text instance.

Scoring recipe

def compute_borda_count(task_scores):
    votes = {model: 0 for model in task_scores}
    for task, scores in task_scores.items():
        ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
        for rank, (model, _) in enumerate(ranked):
            votes[model] += len(ranked) - rank
    return votes

Common pitfalls

• Relying solely on Mean(Task) score, which can be inflated by strong performance on a small subset of benchmarks rather than indicating consistent generalization.
• Ignoring task-specific instruction templates, which are critical for instruction-aware embedding models to perform optimally on MTEB tasks.

Evidence (verbatim from paper)

Ranking on the official MMTEB Leaderboard is determined by the Borda count method. Each task is treated as a preference voter, which gives votes to the models based on their relative performance on the task. The best model obtains the highest number of votes. The model with the highest number of votes across all tasks obtains the highest rank.

Citation

@misc{babakhin2025llamaembednemotron8b,
  title={Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks},
  author={Babakhin et al. (2025)},
  year={2025},
  note={arXiv:2511.07025}
}

• arXiv: 2511.07025