Mteb Airbench Eval

mteb-airbench-eval

Enhancing Lexicon-Based Text Embeddings with Large Language Models — Lei et al. (2025) (arXiv:2501.09749, 2025)

What this evaluates

Evaluates text embedding models on retrieval, reranking, clustering, classification, semantic textual similarity, and summarization tasks. It also assesses out-of-domain generalization on domain-specific question answering and retrieval benchmarks.

Datasets

• MTEB — total 56; splits: test (-1); HF mteb
• AIR-Bench — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures retrieval quality by summing graded relevance scores of retrieved documents, discounted logarithmically by their position in the result list, and normalizing by the ideal DCG.

Input / output format

Input: Query and passage/document pairs (or single texts for non-retrieval tasks) processed by the embedding model to produce fixed-dimensional vectors.

Output: Fixed-dimensional embedding vectors (e.g., 4000-d or 8000-d) or ranked lists of documents based on cosine similarity scores.

Scoring recipe

def compute_ndcg_at_k(relevant_docs, ranked_docs, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(ranked_docs[:k]) if doc in relevant_docs)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0

# For MTEB, apply task-specific metrics (e.g., nDCG@10 for retrieval, accuracy for classification) and average across 56 datasets.

Common pitfalls

• MTEB covers 7 distinct task types, each requiring a different evaluation metric; nDCG@10 only applies to retrieval tasks.
• AIR-Bench test set ground truth is hidden; evaluation must use the specified 24.04 version to ensure consistency.
• The paper explicitly evaluates in a zero-shot setting, omitting in-context learning samples that some baselines use.

Evidence (verbatim from paper)

We evaluate the performance of various embedding models using MTEB and AIR-Bench. MTEB is a comprehensive text embedding benchmark encompassing seven task types across a total of 56 datasets. ... Table 2: QA performance on AIR-Bench 24.04 (English) across different models, where nDCG@10 is used as the metric.

Citation

@misc{lei2025lens,
  title={Enhancing Lexicon-Based Text Embeddings with Large Language Models},
  author={Lei et al. (2025)},
  year={2025},
  note={arXiv:2501.09749}
}

• arXiv: 2501.09749