Mteb Eval

mteb-eval

KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs — Tang et al. (2026) (arXiv:2601.01046, 2026)

What this evaluates

This evaluation probes the ability of decoder-only LLMs to generate high-quality, fixed-dimensional text embeddings without fine-tuning. It measures semantic similarity, information retrieval, classification, clustering, and long-context comprehension across diverse tasks and sequence lengths.

Datasets

• MTEB — total ?; splits: test (-1)
• LoCoV1 — total ?; splits: test (-1)

Metrics

• MTEB Average Score (primary) — range: [0, 1]
  • Task-specific metric (e.g., cosine similarity for STS, accuracy for classification, NDCG@10 for retrieval) computed per dataset, averaged within each of the seven task categories, then averaged across categories for an overall score.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring the quality of ranked document retrieval lists against ground-truth relevant documents.

Input / output format

Input: Text sequences (sentences or documents) fed to decoder-only LLMs. Max length 512 for MTEB; truncated to 1024, 2048, or 4096 tokens for LoCoV1.

Output: Fixed-dimensional text embeddings extracted from internal Key-Value (KV) states of specified transformer layers, without modifying the input sequence.

Scoring recipe

def compute_mteb_avg(datasets_by_category):
    category_scores = []
    for cat, ds_list in datasets_by_category.items():
        task_scores = []
        for ds in ds_list:
            preds = extract_kv_embeddings(ds.texts)
            gold = ds.labels
            task_scores.append(compute_task_metric(preds, gold))
        category_scores.append(mean(task_scores))
    return mean(category_scores)

def compute_locov1_score(retrieval_preds, gold_docs, k=10):
    return ndcg_at_k(retrieval_preds, gold_docs, k)

Common pitfalls

• Assuming the method requires gradient-based fine-tuning; it is explicitly training-free and zero-shot.
• Misinterpreting the 'w/o KV Re-routing' row as the main method; it is an ablation that disables the core re-routing mechanism while keeping the compression prompt.
• Overlooking that MTEB scores are category-averaged task-specific metrics, not a single unified score, and vary by backbone architecture.

Evidence (verbatim from paper)

We evaluate KV-Embedding on two complementary benchmarks. MTEB provides a multi-task assessment across seven categories: STS, Retrieval, Classification, Pair Classification, Clustering, Reranking, and Summarization. To evaluate robustness in long-context scenarios, we use LoCoV1, truncating documents to 1024, 2048, and 4096 tokens. Table 2: Retrieval performance (NDCG@10) on LoCoV1 across different context lengths.

Citation

@misc{tang2026kvembedding,
  title={KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs},
  author={Tang et al. (2026)},
  year={2026},
  note={arXiv:2601.01046}
}

• arXiv: 2601.01046