Mteb Subset Eval

mteb-subset-eval

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders — BehnamGhader et al. (2024) (arXiv:2404.05961, 2024)

What this evaluates

Evaluates text embedding models across diverse semantic tasks including retrieval, reranking, clustering, pair classification, classification, and semantic textual similarity to measure the quality of dense vector representations.

Datasets

• MTEB (15-task subset) — total ?; splits: test (-1)

Metrics

• MTEB average score (primary) — range: [0, 1]
  • Mean of task-specific evaluation scores across the 15 selected tasks. Each task uses its standard metric (e.g., NDCG@10 for retrieval, accuracy for classification, cosine similarity for STS).

Input / output format

Input: Text inputs vary by task: query-document pairs for retrieval, sentence pairs for STS and pair classification, single documents for clustering and classification.

Output: Normalized dense vector embeddings for each text input.

Scoring recipe

scores = []
for task in mteb_subset:
    preds = model.encode(task.documents)
    gold = task.labels
    scores.append(task_metric(preds, gold))
return sum(scores) / len(scores)

Common pitfalls

• MTEB tasks require different preprocessing and evaluation protocols; applying a single metric across all tasks yields invalid results.
• The subset explicitly skips the SummEval category, so scores are not comparable to full MTEB leaderboards.
• Embeddings must be L2-normalized before computing cosine similarity for STS and retrieval tasks.

Evidence (verbatim from paper)

When evaluating on MTEB, we use the same instructions as Wang et al. (2023). The list of instructions for each task is listed in Table 10. To make sure that our ablation and analyses are not biased towards one specific category or task, this subset includes tasks from each category with almost the same proportion compared to the full MTEB.

Citation

@misc{behnamghader2024llm2vec,
  title={LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders},
  author={BehnamGhader et al. (2024)},
  year={2024},
  note={arXiv:2404.05961}
}

• arXiv: 2404.05961