Embeddinggemma Eval

embeddinggemma-eval

EmbeddingGemma: Powerful and Lightweight Text Representations — Henrique Schechter Vera et al. (2025) (arXiv:2509.20354, 2025)

What this evaluates

Evaluates the quality of text embeddings across diverse tasks including retrieval, classification, clustering, and semantic similarity. It probes multilingual, cross-lingual, and code understanding capabilities, measuring how well dense vector representations capture semantic relationships for downstream applications.

Datasets

• MTEB (Massive Text Embedding Benchmark) — total 162; splits: test (-1)
• XOR-Retrieve — total ?; splits: test (-1)
• XTREME-UP — total ?; splits: test (-1)

Metrics

• MTEB Task Mean (primary) — range: [0, 1]
  • Average of normalized scores across all individual MTEB tasks. Scores are typically normalized to [0, 1] per task before averaging.
• MTEB Type Mean — range: [0, 1]
  • Average of normalized scores grouped by task type (e.g., Retrieval, Classification, Clustering, STS).
• Borda rank — range: rank
  • Ranking metric computed by assigning points based on task-level performance across all models, then summing to produce an aggregate rank.
• Recall@5kt — range: [0, 1]
  • Fraction of queries where the correct passage is retrieved within the top 5,000 candidates.
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank at 10, averaging the inverse rank of the first correct passage across queries.

Input / output format

Input: Text pairs (query, passage) or single documents, formatted with task-specific prompt instructions from the model card. Context length is typically 512 tokens, extended to 1024 or 2048 for long-context tasks.

Output: Dense embedding vectors, defaulting to 768 dimensions but also evaluated at 512, 256, and 128 dimensions.

Scoring recipe

def compute_mteb_scores(predictions, gold, task_type):
    if task_type in ['retrieval', 'reranking']:
        return compute_ranking_metrics(predictions, gold, k=10)
    elif task_type in ['classification', 'multilabel']:
        return compute_accuracy(predictions, gold)
    elif task_type == 'clustering':
        return compute_clustering_score(predictions, gold)
    elif task_type == 'sts':
        return pearson_spearson_correlation(predictions, gold)
    return normalized_score

def aggregate_scores(task_scores):
    task_mean = mean(task_scores)
    type_mean = mean(group_by_type(task_scores))
    return task_mean, type_mean

Common pitfalls

• Models trained on more than 25% of the MTEB data are excluded to mitigate potential overfitting; failing to check training data composition can lead to invalid comparisons.
• Prompt instructions must strictly follow the model card; using default or mismatched prompts significantly degrades performance.
• Context length varies by task (512 vs 1024/2048); using a fixed context length across all tasks may underperform on long-context benchmarks.
• Embedding dimensionality heavily impacts scores; comparing 768d results directly with 128d results without noting the trade-off is misleading.

Evidence (verbatim from paper)

We also show results on XOR-Retrieve and XTREME-UP, reporting Recall@5kt and MRR@10 respectively. EmbeddingGemma achieves the #1 rank and highest overall performance on the MTEB multilingual, English, and code leaderboards across models under 500M parameters, with a significant lead over all previous top performing models on each of the metrics summarizing aggregate performance across tasks (Task Mean, Task Type Mean, and Borda rank).

Citation

@misc{vera2025embeddinggemma,
  title={EmbeddingGemma: Powerful and Lightweight Text Representations},
  author={Henrique Schechter Vera et al. (2025)},
  year={2025},
  note={arXiv:2509.20354}
}

• arXiv: 2509.20354