Universal Embedder Eval

universal-embedder-eval

Language Models are Universal Embedders — Xin Zhang et al. (2023) (arXiv:2310.08232, 2023)

What this evaluates

Evaluates the cross-lingual and cross-domain generalization of decoder-based language models finetuned via contrastive learning on English data. Probes the model's ability to generate unified embeddings for natural language and code retrieval, semantic textual similarity, and intent classification across diverse languages and domains.

Datasets

• MTEB — total ?; splits: test (-1)
• CodeSearchNet — total ?; splits: test (-1)
• Multi-CPR — total ?; splits: test (-1)
• MASSIVE — total ?; splits: test (-1)
• STS-17 & STS-22 — total ?; splits: test (-1)
• MIRACL — total ?; splits: test (-1)
• BUCC — total ?; splits: test (-1)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Rank correlation between cosine similarity of sentence embeddings and human-annotated STS scores (1–5).
• MTEB average score — range: [0, 1]
  • Arithmetic mean of task-specific scores (accuracy, Spearman correlation, or Recall@k/MRR@k) across all 56 datasets in the MTEB benchmark.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10 for multilingual retrieval tasks.
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank at top 10 for domain-specific retrieval benchmarks.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted intent labels in linear-probe classification setups.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for cross-lingual sentence pair retrieval.

Input / output format

Input: Sentence pairs (query, document) for retrieval; sentence pairs for semantic similarity; single sentences for classification.

Output: Continuous embedding vectors; for classification, predicted class labels.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'classification':
        return accuracy_score(golds, predictions)
    elif task_type == 'sts':
        return spearmanr(cosine_similarity(predictions), golds).correlation
    elif task_type == 'retrieval':
        return recall_at_k(golds, predictions, k=100)
    return mean([compute_metrics(p, g, t) for p, g, t in zip(preds, golds, tasks)])

Common pitfalls

• Averaging heterogeneous metrics (accuracy, Spearman, Recall) across 56 datasets can mask task-specific weaknesses.
• Evaluating on languages outside the model's pre-training vocabulary without controlling for parameter scaling.
• Comparing against proprietary APIs (e.g., Cohere) that may have been trained on or exposed to evaluation data.

Evidence (verbatim from paper)

Following the STS evaluation protocol of MTEB, we use the Spearman correlation between the cosine similarity of the sentence embeddings and the human-annotated scores (from 1 to 5) as the main metric.

Citation

@misc{zhang2023language,
  title={Language Models are Universal Embedders},
  author={Xin Zhang et al. (2023)},
  year={2023},
  note={arXiv:2310.08232}
}

• arXiv: 2310.08232