Gte Eval

gte-eval

Towards General Text Embeddings with Multi-stage Contrastive Learning — Li et al. (2023) (arXiv:2308.03281, 2023)

What this evaluates

Evaluates the cross-task generalization and retrieval quality of a general-purpose text embedding model across classification, retrieval, clustering, reranking, semantic similarity, summarization, and code search tasks. It measures how well zero-shot and unsupervised embeddings transfer to diverse downstream benchmarks without task-specific fine-tuning.

Datasets

• SST-2 — total ?; splits: test (-1)
• BEIR — total ?; splits: test (-1)
• MTEB (English subset) — total 56; splits: test (-1)
• CodeSearchNet — total ?; splits: dev+test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring the quality of ranked retrieval results by penalizing relevant documents appearing lower in the list.
• accuracy — range: [0, 1]
  • Proportion of correctly classified instances in zero-shot text classification, computed by matching input embeddings to verbalized label embeddings.
• v-measure — range: [0, 1]
  • Harmonic mean of homogeneity and completeness, used for evaluating text clustering quality.
• Spearman coefficient — range: [-1, 1]
  • Rank correlation coefficient measuring the monotonic relationship between predicted similarity scores and human judgments for STS and reranking tasks.

Input / output format

Input: Text pairs (query, document) or single texts for classification. For classification, inputs are paired with verbalized label texts (e.g., 'positive'/'negative' or fuzzy prompt templates). For retrieval, inputs are queries against a large candidate corpus.

Output: Embedding vectors (typically [CLS] token or mean pooling) used to compute similarity scores via dot product or cosine similarity. Final output is a ranked list of candidates or a classification label.

Scoring recipe

query_emb = model.encode(query)
candidate_embs = [model.encode(c) for c in candidates]
scores = [np.dot(query_emb, e) for e in candidate_embs]  # or cosine
ranked = np.argsort(scores)[::-1]
if task == 'classification':
  metric = 1.0 if ranked[0] == gold_idx else 0.0
else:
  metric = compute_ndcg(ranked, relevance_labels, k=10)
return mean(metric across instances)

Common pitfalls

• Models use different similarity functions (dot product vs cosine) which drastically changes scores and breaks direct comparison.
• Comparing models of vastly different parameter sizes without normalizing for inference speed or computational cost.
• Confusing unsupervised pre-training results with supervised fine-tuning results when reading benchmark tables.

Evidence (verbatim from paper)

The evaluation metrics employed in MTEB are accuracy, v-measure, average precision, MAP, nDCG@10, and Spearman coefficients, respectively. ... We use BEIR (Thakur et al., 2021) as our evaluation benchmark for zero-shot unsupervised text retrieval.

Citation

@misc{li2023gte,
  title={Towards General Text Embeddings with Multi-stage Contrastive Learning},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2308.03281}
}

• arXiv: 2308.03281