Mteb Beir Miracl Eval

mteb-beir-miracl-eval

Improving Text Embeddings with Large Language Models — Wang et al. (2024) (arXiv:2401.00368, 2024)

What this evaluates

Evaluates text embedding models across diverse NLP tasks including classification, clustering, semantic textual similarity, reranking, and information retrieval. It specifically probes multilingual retrieval capabilities and measures how effectively synthetic data generation improves embedding quality without relying on labeled supervision.

Datasets

• MTEB (English subset) — total ?; splits: test (-1); HF mteb
• BEIR (Retrieval) — total ?; splits: test (-1)
• MIRACL — total ?; splits: dev (-1)

Metrics

• MTEB official metrics (primary) — range: percent
  • Task-specific metrics (e.g., accuracy, F1, nDCG@10, Spearman correlation) reported per category and averaged across 56 English datasets. Normalization and averaging follow the official MTEB benchmark specification.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, computed on the MIRACL dev set across 18 languages to measure retrieval ranking quality.

Input / output format

Input: Text queries and documents (or task-specific instructions/pairs), truncated to the first 512 tokens for efficiency.

Output: Dense vector embeddings for each text instance, or ranked lists/scores depending on the downstream task category.

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'retrieval':
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'classification':
        return accuracy(predictions, gold)
    elif task_type == 'sts':
        return spearman_correlation(predictions, gold)
    # Follow MTEB official metric mapping for other categories
    return official_metric(predictions, gold)

def aggregate_mteb(all_task_scores):
    return sum(all_task_scores) / len(all_task_scores)

Common pitfalls

• The evaluation truncates all inputs to 512 tokens for efficiency, which may unfairly penalize models capable of leveraging longer contexts.
• MTEB averages metrics across heterogeneous task categories; readers must apply the official MTEB normalization and averaging procedure rather than naive arithmetic means.
• Commercial model comparisons are limited due to lack of transparency in their training data and architectures, making direct performance attribution difficult.

Evidence (verbatim from paper)

We evaluate the trained model on the MTEB benchmark. Note that the retrieval category in MTEB corresponds to the 15 publicly available datasets in the BEIR benchmark. Official metrics are reported for each category. ... Table 3: nDCG@10 on the dev set of the MIRACL dataset for both high-resource and low-resource languages.

Citation

@misc{wang2024improving,
  title={Improving Text Embeddings with Large Language Models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.00368}
}

• arXiv: 2401.00368