C Mteb Sentence Embed Eval

c-mteb-sentence-embed-eval

Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition — Duan et al. (2025) (arXiv:2502.20726, 2025)

What this evaluates

Evaluates the zero-shot quality of sentence and word embeddings from decoder-only LLMs on downstream tasks without additional training. Probes capabilities in text classification, clustering, retrieval, semantic similarity, and polysemous word disambiguation.

Datasets

• C-MTEB — total ?; splits: test (-1)
• SLPWC (C-SEM) — total 300; splits: test (300)
• WSD — total ?; splits: test (-1)

Metrics

• Average C-MTEB normalized score (primary) — range: [0, 1]
  • Task-specific metrics (e.g., accuracy, cosine similarity) are computed per C-MTEB task, normalized to [0,1] following standard C-MTEB protocol, and averaged across six categories: classification, pair classification, clustering, retrieval, STS, and reranking.
• accuracy — range: percent
  • Computed on SLPWC and WSD datasets by selecting the option with the maximum/minimum embedding distance (Cosine or Euclidean) depending on the task formulation, then calculating the proportion of correct predictions over the test set.

Input / output format

Input: Text sequences for sentence embedding tasks; target words within contextual sentences for word embedding tasks.

Output: Dense vector embeddings (sentence-level or token-level) used for downstream similarity/classification tasks. No specific text generation format required.

Scoring recipe

def score_c_mteb(embeddings, gold):
    task_scores = []
    for task in C_MTEB_TASKS:
        preds = compute_task_metric(embeddings[task], gold[task])
        task_scores.append(normalize_to_0_1(preds))
    return mean(task_scores)

def score_word_emb(embeddings, gold):
    correct = 0
    for item in dataset:
        dists = [cosine_sim(target_emb, opt_emb) for opt in options]
        pred = argmax(dists)
        if pred == gold: correct += 1
    return correct / len(dataset)

Common pitfalls

• Confusing ReBA (which uses backward attention weights to refine embeddings) with simple text repetition (Echo), which actually degrades word-level performance.
• Assuming more repetitions (e.g., 3x) improve results; the paper shows 2x is sufficient and additional repetitions yield marginal gains.
• Using mean pooling instead of last-token pooling; last-pooling yields substantially higher scores for ReBA on C-MTEB.

Evidence (verbatim from paper)

We primarily evaluated GPT-2-Chinese (we call it GPT-2 for simplicity) and LLaMA-2-Chinese-7B(LLaMA-2, for simplicity) on C-MTEB, comparing the performance of our method with traditional encoding approaches under different pooling strategies. Across nearly all tasks, our method demonstrated significant improvements over traditional methods. We evaluate performance based on accuracy and adapt it into a 4-choice format to suit our algorithm.

Citation

@misc{duan2025reba,
  title={Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition},
  author={Duan et al. (2025)},
  year={2025},
  note={arXiv:2502.20726}
}

• arXiv: 2502.20726