Gemini Embedding Eval

gemini-embedding-eval

Gemini Embedding: Generalizable Embeddings from Gemini — Lee et al. (2025) (arXiv:2503.07891, 2025)

What this evaluates

Evaluates the quality of multilingual and code-aware text embeddings across diverse tasks including retrieval, classification, clustering, and cross-lingual matching. It probes the model's ability to generate generalizable representations that perform well across 250+ languages, English, and programming languages.

Datasets

• MMTEB — total 164; splits: test (-1)
• XTREME-UP — total ?; splits: test (-1)
• XOR-Retrieve — total ?; splits: test (-1)

Metrics

• Task Mean (primary) — range: [0, 1] or percent
  • Equal weighting of all individual task scores across the benchmark.
• Task Type Mean — range: [0, 1] or percent
  • Equal weighting of all task type scores (e.g., Retrieval, Classification, STS).
• Borda rank — range: integer
  • Official leaderboard ranking metric based on aggregate task performance.
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank at cutoff 10, used for cross-lingual retrieval tasks.

Input / output format

Input: Text queries and passages/documents in various languages (including English, 250+ languages, and code), optionally with task-specific prompts.

Output: Dense vector embeddings for each text instance, used to compute cosine similarity for ranking/retrieval tasks.

Scoring recipe

def compute_metrics(embeddings, queries, passages, gold_labels, task_type):
    scores = cosine_similarity(queries, passages)
    if task_type in ['retrieval', 'reranking']:
        ranks = argsort(scores, descending=True)
        return compute_mrr(ranks, gold_labels, k=10)
    elif task_type == 'classification':
        return accuracy(scores, gold_labels)
    # ... other task-specific scorers (STS, clustering, etc.)
    return task_score

task_scores = [compute_metrics(e, q, p, g, t) for t, e, q, p, g in tasks]
task_mean = mean(task_scores)
task_type_mean = mean(mean(task_scores_by_type))
borda_rank = rank_by_score(task_mean)

Common pitfalls

• Confusing Task Mean (equal weighting of all tasks) with Task Type Mean (equal weighting of task types), which can yield different model rankings.
• MTEB(Code) leaderboard results exclude 4 specific tasks (CodeFeedbackMT, CodeFeedbackST, StackOverflowQA, SyntheticText2SQL); reporting on all 12 without noting the exclusion misrepresents standing.
• Cross-lingual retrieval benchmarks (XTREME-UP, XOR-Retrieve) evaluate zero-shot retrieval without translation, which heavily penalizes models lacking shared multilingual alignment.

Evidence (verbatim from paper)

Gemini Embedding establishes a new state-of-the-art in performance, achieving the highest overall performance on the MTEB(Multilingual) leaderboard (March 10th, 2025) with a substantial performance lead over all previous top performing models on each of the overall metrics summarizing aggregate performance across tasks: Task Mean (equal weighting of all tasks): 68.32, Task Type Mean (equal weighting of all task types): 59.64, and Borda rank #1 (official leaderboard ranking metric).

Citation

@misc{lee2025geminiembedding,
  title={Gemini Embedding: Generalizable Embeddings from Gemini},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2503.07891}
}

• arXiv: 2503.07891