Mmteb Eval

mmteb-eval

MMTEB: Massive Multilingual Text Embedding Benchmark — Enevoldsen et al. (2025) (arXiv:2502.13595, 2025)

What this evaluates

Evaluates the quality of multilingual text embeddings across diverse tasks and languages. It probes capabilities like semantic similarity, classification, retrieval, and multilingual alignment.

Datasets

• MTEB(Multilingual) — total 132; splits: test (-1); repo https://github.com/embeddings-benchmark/mteb
• MTEB(Europe) — total 74; splits: test (-1); repo https://github.com/embeddings-benchmark/mteb
• MTEB(Indic) — total 23; splits: test (-1); repo https://github.com/embeddings-benchmark/mteb

Metrics

• Borda count (primary) — range: integer (higher is better)
  • Each task ranks all evaluated models. Models receive points based on their rank (e.g., highest score gets N points, lowest gets 1). Points are summed across all tasks to yield a total score. Ties are resolved via the tournament Borda count method.
• Average task score — range: percent or [0, 1]
  • The arithmetic mean of task-specific performance metrics (e.g., accuracy, cosine similarity, F1) computed across all tasks or grouped by category.

Input / output format

Input: Text pairs (for classification, STS, bitext mining) or query-document pairs (for retrieval, reranking) fed into a text embedding model.

Output: Fixed-dimensional embedding vectors for each input text.

Scoring recipe

def evaluate(models, tasks):
    task_scores = {t: [] for t in tasks}
    for model in models:
        for t in tasks:
            task_scores[t].append(compute_task_metric(model, t))
    avg_scores = {m: mean([s for s in task_scores.values()]) for m in models}
    borda_scores = {m: 0 for m in models}
    for t in tasks:
        ranked = sort_descending(task_scores[t])
        for rank, m in enumerate(ranked):
            borda_scores[m] += len(ranked) - rank
    return avg_scores, borda_scores

Common pitfalls

• Averaging raw task scores without normalization can skew results, as different tasks use different metrics (e.g., accuracy vs. cosine similarity).
• Ignoring the Borda count aggregation method for final model ranking, which is explicitly designed to be more robust than simple score averaging.
• Overlooking limited model support for newer tasks like Instruction Retrieval, which are excluded from category averages in reported results.

Evidence (verbatim from paper)

We compute model ranks using the Borda count method (Colombo et al., [2022]), derived from social choice theory. This method, which is also employed in election systems based on preference ranking, has been shown to be more robust for comparing NLP systems. To compute this score, we consider each task as a preference voter voting for each model, and scores are aggregated according to the Borda Count method. In the case of ties, we use the tournament Borda count method.

Citation

@misc{enevoldsen2025mmteb,
  title={MMTEB: Massive Multilingual Text Embedding Benchmark},
  author={Enevoldsen et al. (2025)},
  year={2025},
  note={arXiv:2502.13595}
}

• arXiv: 2502.13595