Mteb Echo Eval

mteb-echo-eval

Repetition Improves Language Model Embeddings — Springer et al. (2024) (arXiv:2402.15449, 2024)

What this evaluates

Evaluates the quality of text embeddings extracted from autoregressive language models in zero-shot and fine-tuned settings across a broad suite of downstream NLP tasks including classification, clustering, retrieval, and semantic textual similarity.

Datasets

• MTEB — total 56; splits: test (-1)

Metrics

• Average MTEB Score (primary) — range: [0, 100]
  • Task-specific metrics (e.g., accuracy, NDCG@10, Pearson correlation) computed per dataset, then averaged across all 56 datasets. Higher is better.

Input / output format

Input: Text input $S$, optionally wrapped in a task-specific prompt (e.g., 'Rewrite the following paragraph: $S$. The rewritten paragraph: $S$' for zero-shot, or 'Query: $S$' / 'Document: $S$' for fine-tuning).

Output: Fixed-dimensional embedding vector extracted from the model's hidden states using mean pooling or last-token pooling.

Scoring recipe

def compute_mteb_score(predictions, gold, task_type):
    if task_type == 'classification':
        return accuracy(predictions, gold)
    elif task_type in ['reranking', 'retrieval']:
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'sts':
        return pearson_correlation(predictions, gold)
    # ... other task-specific metrics ...
    return average_across_datasets(task_scores)

Common pitfalls

• Echo embeddings inherently require twice the inference compute of classical embeddings due to input repetition; failing to match compute (e.g., by halving input length) leads to unfair comparisons.
• Pooling strategy significantly impacts zero-shot performance (mean pooling is required), but last-token pooling often performs better in fine-tuned settings.
• Casting causal attention to bidirectional without fine-tuning typically harms performance, except for models like Mistral-7B with non-standard pretraining.

Evidence (verbatim from paper)

Our main evaluation dataset is the English-language subset of the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2022). MTEB is a collection of 56 datasets that are grouped into different embedding tasks: classification, clustering, pair classification, reranking, retrieval, sentence similarity (STS), and summarization, with the goal of evaluating embeddings broadly. ... Table 2: Role of scale and base model in the zero-shot setting: Average MTEB score (56 datasets) for LLaMA-7B-2-Instruct and S-LLaMA-1.3B.

Citation

@misc{springer2024repetition,
  title={Repetition Improves Language Model Embeddings},
  author={Springer et al. (2024)},
  year={2024},
  note={arXiv:2402.15449}
}

• arXiv: 2402.15449