Mteb Clustering Eval

mteb-clustering-eval

Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning — Benedikt Roth et al. (2025) (arXiv:2507.22729, 2025)

What this evaluates

Evaluates the quality of text embeddings for document clustering by measuring how well the embeddings group semantically related sentences together. It probes the model's ability to capture semantic similarity without task-specific fine-tuning or with lightweight adaptation.

Datasets

• MTEB v1.38.30 English Clustering — total ?; splits: test (-1)

Metrics

• clustering accuracy (primary) — range: percent
  • Macro-average clustering accuracy (vmeasure) computed per dataset and averaged across the 21 English clustering datasets in MTEB v1.38.30. Scores are reported as single-run values without confidence intervals.

Input / output format

Input: Raw text sentences, optionally wrapped with task-specific prompts (e.g., CCW, CCP, or classification prompts).

Output: Fixed-dimensional embedding vectors derived from the LLM's hidden states using either mean pooling or last-token pooling.

Scoring recipe

scores = []
for dataset in mteb_english_clustering_datasets:
    embeddings = model.encode(dataset.documents, pooling='mean')
    clusters = agglomerative_cluster(embeddings)
    scores.append(vmeasure_score(clusters, dataset.labels))
final_metric = mean(scores) * 100  # reported as percent

Common pitfalls

• MTEB protocol mandates single-run scores without confidence intervals or variance estimates for direct comparability.
• Pooling strategy drastically changes results; mean pooling generally outperforms last-token in zero-shot, but last-token benefits more from prompt engineering and fine-tuning.
• Prompt formatting must be applied consistently; wrapping prompts around input text yields different results than appending them.

Evidence (verbatim from paper)

The zero-shot baseline was obtained by comparing three token aggregation strategies across Llama-3.2-1B and Qwen3-0.6B. As shown in Table 3, mean pooling delivers the highest average clustering accuracy, outperforming last-token pooling by approximately $12%$ and EOS-token pooling by around $20%$ across both models.

Citation

@misc{roth2025resourceefficient,
  title={Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning},
  author={Benedikt Roth et al. (2025)},
  year={2025},
  note={arXiv:2507.22729}
}

• arXiv: 2507.22729