Mteb English Eval

mteb-english-eval

Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models — Lin et al. (2025) (arXiv:2507.23386, 2025)

What this evaluates

Evaluates the ability of decoder-only LLMs to generate universal text embeddings across diverse natural language processing tasks. It probes retrieval, reranking, clustering, classification, pair classification, semantic textual similarity, and summarization capabilities using standardized benchmark datasets.

Datasets

• MTEB (English subset) — total ?; splits: test (-1)

Metrics

• nDCG@10 — range: [0, 1]
  • Normalized discounted cumulative gain at rank 10, used for retrieval tasks.
• MAP — range: [0, 1]
  • Mean Average Precision, used for reranking tasks.
• V-measure — range: [0, 1]
  • V-measure for clustering tasks, measuring homogeneity and completeness.
• AP — range: [0, 1]
  • Average Precision for pair classification tasks.
• Accuracy — range: [0, 1]
  • Classification accuracy for classification tasks.
• Spearman correlation — range: [-1, 1]
  • Spearman rank correlation for STS and summarization tasks.
• Average score (primary) — range: [0, 1]
  • Macro-average of task-specific metrics across all 56 datasets.

Input / output format

Input: Text inputs are first processed by a BERT-style encoder to generate a single Contextual token, which is prepended to the LLM input sequence. The model receives the concatenated sequence of the Contextual token and the original text tokens.

Output: The model outputs hidden states for all tokens. The final text embedding is constructed by concatenating the last hidden states of the Contextual token and the EOS token.

Scoring recipe

def score(predictions, gold, task_type):
    if task_type == 'Retrieval': return ndcg_at_k(gold, predictions, k=10)
    elif task_type == 'Reranking': return map_score(gold, predictions)
    elif task_type == 'Clustering': return v_measure(gold, predictions)
    elif task_type == 'PairClassification': return ap_score(gold, predictions)
    elif task_type == 'Classification': return accuracy(gold, predictions)
    elif task_type in ['STS', 'Summarization']: return spearman_corr(gold, predictions)
    return None

Common pitfalls

• Training on non-public or synthetic data inflates MTEB scores but harms generalization to unseen non-retrieval tasks.
• Using standard last-token pooling introduces recency bias; the protocol explicitly requires concatenating Contextual and EOS token states to mitigate this.
• Comparing against methods that use bidirectional attention or in-context learning examples without accounting for increased sequence length and computational overhead.

Evidence (verbatim from paper)

The main evaluation metrics are nDCG@10, MAP, V-measure (V-meas.), average precision (AP), accuracy (Acc.), and Spearman correlation (Spear., both for STS and Summ.), respectively.

Citation

@misc{lin2025causal2vec,
  title={Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2507.23386}
}

• arXiv: 2507.23386