Mteb Eng V2 Eval

mteb-eng-v2-eval

LGAI-EMBEDDING-Preview Technical Report — Choi et al. (2025) (arXiv:2506.07438, 2025)

What this evaluates

Evaluates the semantic discrimination and generalization capabilities of text embedding models across diverse NLP tasks including retrieval, classification, clustering, and semantic similarity. It uses a zero-shot English-only benchmark to measure performance without task-specific fine-tuning.

Datasets

• MTEB(eng, v2) — total ?; splits: test (-1)

Metrics

• average score across all tasks (primary) — range: [0, 1] | percent
  • Arithmetic mean of task-specific metric scores (e.g., accuracy, nDCG@10, cosine similarity) across all 41 tasks in the benchmark.
• Borda count ranking — range: rank
  • Each task acts as a voter assigning preference rankings to models. Final rankings are computed by aggregating these votes, with ties resolved using the tournament Borda count variant.

Input / output format

Input: Text pairs, queries, or documents depending on the specific MTEB task (e.g., query-document pairs for retrieval, sentence pairs for STS, labeled instances for classification).

Output: Fixed-dimensional dense vector embeddings for each input text instance.

Scoring recipe

task_scores = [run_task_on_model(task) for task in mteb_eng_v2_tasks]
avg_score = sum(task_scores) / len(task_scores)
category_avg = {cat: mean(scores_in_cat) for cat, scores_in_cat in group_by_category(task_scores)}
for task in mteb_eng_v2_tasks:
    rankings = rank_models_by(task_scores[task])
    aggregate_votes += rankings
final_ranking = apply_tournament_borda(aggregate_votes)

Common pitfalls

• The benchmark uses zero-shot evaluation, so models should not be fine-tuned on the test splits.
• Task-specific metrics vary (e.g., accuracy for classification, nDCG@10 for retrieval), so the aggregate score is a simple mean of normalized task scores, not a single unified metric.
• Ties in Borda count rankings are resolved using a tournament variant, which may differ from standard Borda implementations.

Evidence (verbatim from paper)

The MTEB(eng, v2) benchmark was newly introduced this year as an English-only evaluation suite designed to improve computational efficiency and reduce inter-task correlation through optimized task selection. In this work, we target the MTEB(eng, v2) benchmark for evaluating our model’s performance. In addition, the Borda count ranking method is adopted for model comparison, following the evaluation protocol of the MTEB(eng, v2) benchmark, as it provides robust aggregation of performance across diverse tasks. For evaluation, average scores across all tasks, per-category averages, and category-weighted averages are reported. In the Borda count framework, each task is treated as a voter assigning preference rankings to models. Final rankings are computed by aggregating these votes, and in the event of ties, the tournament Borda count variant is applied to ensure ranking stability.

Citation

@misc{choi2025lgaiembeddingpreview,
  title={LGAI-EMBEDDING-Preview Technical Report},
  author={Choi et al. (2025)},
  year={2025},
  note={arXiv:2506.07438}
}

• arXiv: 2506.07438