C Mteb Eval

c-mteb-eval

C-Pack: Packed Resources For General Chinese Embeddings — Shitao Xiao et al. (arXiv:2309.07597, 2023)

What this evaluates

Evaluates the generalization and effectiveness of Chinese text embedding models across six core NLP tasks: retrieval, semantic textual similarity (STS), pair classification, single-label classification, re-ranking, and clustering. It measures how well dense vector representations capture semantic relationships for diverse downstream applications.

Datasets

• C-MTEB — total 35; splits: test (-1); repo https://github.com/FlagOpen/FlagEmbedding

Metrics

• Average Performance (primary) — range: percent
  • The arithmetic mean of task-specific evaluation scores across the six C-MTEB tasks (Retrieval, STS, Pair CLF, CLF, Re-rank, Cluster). Task-specific scores are computed using standard protocols (e.g., nDCG@10 for retrieval, Spearman correlation for STS, accuracy for classification) and normalized to a 0-100 scale before averaging.

Input / output format

Input: Query-document pairs for retrieval/re-ranking, sentence pairs for STS, and instruction-prefixed texts for classification/clustering tasks.

Output: Fixed-dimensional embedding vectors (384, 768, or 1024 dimensions). Task-specific predictions (predicted labels, similarity scores, or ranked document lists) are derived from these embeddings during evaluation.

Scoring recipe

def compute_c_mteb_avg(tasks, embeddings):
    task_scores = []
    for task in tasks:
        if task.type == 'retrieval':
            score = compute_ndcg_at_k(task.gold, embeddings, k=10)
        elif task.type == 'sts':
            score = compute_spearman_correlation(task.gold, embeddings)
        elif task.type in ['pair_clf', 'clf']:
            score = compute_accuracy(task.gold, embeddings)
        elif task.type == 're_rank':
            score = compute_ndcg_at_k(task.gold, embeddings, k=10)
        elif task.type == 'cluster':
            score = compute_v_measure(task.gold, embeddings)
        task_scores.append(score)
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Task-specific metrics vary significantly (e.g., nDCG@10 for retrieval vs. Spearman for STS); averaging requires consistent normalization to a 0-100 scale per task.
• Instruction formatting during fine-tuning critically impacts performance; omitting instructions or using inconsistent prompts leads to substantially lower scores.
• Batch size during contrastive pre-training heavily influences retrieval quality; smaller batches yield suboptimal embeddings due to fewer in-batch negatives.

Evidence (verbatim from paper)

First, our models outperform existing Chinese text embeddings by large margins. There is not only an overwhelming advantage in terms of the average performance, but also notable improvements for the majority of tasks in C-MTEB.

Citation

@misc{xiao2023cpack,
  title={C-Pack: Packed Resources For General Chinese Embeddings},
  author={Shitao Xiao et al.},
  year={2023},
  note={arXiv:2309.07597}
}

• arXiv: 2309.07597