Mteb Mmteb Retrieval Eval

mteb-mmteb-retrieval-eval

jina-embeddings-v5-text: Task-Targeted Embedding Distillation — Akram et al. (2026) (arXiv:2602.15547, 2026)

What this evaluates

Evaluates text embedding models across retrieval, semantic similarity, clustering, classification, and reranking tasks. It probes the ability of compact, distillation-trained models to generalize across multilingual corpora, long documents, and enterprise-scale retrieval benchmarks.

Datasets

• MTEB (English v2) — total ?; splits: test (-1)
• MMTEB (Multilingual v2) — total ?; splits: test (-1)
• RTEB (Multilingual) — total ?; splits: test (-1)
• BEIR — total ?; splits: test (-1)
• LongEmbed — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by summing graded relevance values of results, discounted logarithmically by their position in the result list, then normalized by the ideal DCG.
• Spearman correlation — range: [-1, 1]
  • Rank-based correlation coefficient used for STS and summarization tasks. Measures the monotonic relationship between predicted similarity scores and human/ground-truth ratings.
• V-measure — range: [0, 1]
  • Harmonic mean of homogeneity and completeness (V = 2hc/(h+c)). Evaluates clustering quality by measuring cluster purity and class concentration.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances in classification tasks.
• Precision — range: [0, 1]
  • Proportion of true positive predictions among all positive predictions in reranking tasks.

Input / output format

Input: Text strings (queries, documents, or sentence pairs) provided as input to the embedding model.

Output: Fixed-dimensional embedding vectors (e.g., 768 or 1024 dimensions) for each input text.

Scoring recipe

def compute_ndcg_at_k(embeddings_q, embeddings_d, relevance_scores, k=10):
    scores = np.dot(embeddings_q, embeddings_d.T)
    ranked_indices = np.argsort(-scores, axis=1)[:, :k]
    dcg = np.sum(relevance_scores[0, ranked_indices] / np.log2(np.arange(2, k+2)))
    ideal = np.sum(np.sort(relevance_scores, axis=1)[:, -k:][::-1] / np.log2(np.arange(2, k+2)))
    return np.mean(dcg / ideal)

# For STS/Sum: use scipy.stats.spearmanr(pred_scores, gold_scores).correlation
# For Clustering: use sklearn.metrics.v_measure_score(labels_true, labels_pred)
# For Class/Rerank: use sklearn.metrics.accuracy_score or precision_score

Common pitfalls

• Passkey and Needle tasks use nDCG@1 instead of the standard nDCG@10.
• Some leaderboard scores are marked with † indicating partial self-evaluation, which may differ from official benchmark scripts.
• Instruction-tuning strategy (task-specific vs. category-level prefixes) significantly impacts Qwen3-0.6B performance, especially on classification.

Evidence (verbatim from paper)

All retrieval tasks were evaluated using nDCG@10, except for Passkey and Needle, which used nDCG@1. For semantic textual similarity (STS) and summarization tasks, we calculated the Spearman correlation coefficient. For clustering tasks, we used the V-measure666Specifically, the scikit-learn implementationPedregosa et al. ([2011]): the harmonic mean of homogeneity and completeness, $V=\frac{2hc}{h+c}$. Homogeneity measures cluster purity (each cluster contains mostly one true class), while completeness measures class concentration (each true class is mostly assigned to a single cluster). to evaluate the quality of the embeddings. Classification and reranking tasks were evaluated using accuracy and precision metrics.

Citation

@misc{akram2026jinaembeddingsv5,
  title={jina-embeddings-v5-text: Task-Targeted Embedding Distillation},
  author={Akram et al. (2026)},
  year={2026},
  note={arXiv:2602.15547}
}

• arXiv: 2602.15547