Mteb Retrieval Eval

mteb-retrieval-eval

Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores — Lu et al. (2024) (arXiv:2408.11868, 2024)

What this evaluates

Evaluates text embedding models on information retrieval tasks using the MTEB benchmark and a custom e-commerce Q&A dataset. It measures ranking quality via nDCG and mAP, and assesses similarity distribution calibration via AUPRC on a held-out set with single-relevant-passage queries.

Datasets

• MTEB Retrieval — total ?; splits: test (-1)
• E-commerce Q&A — total 26; splits: train (1040), test (546)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computed as DCG@10 divided by the ideal DCG@10 for the same set of labels. DCG sums (2^rel_i - 1) / log2(i+2) for the top-10 results.
• mAP@10 — range: [0, 1]
  • Mean Average Precision at rank 10. Computes the average precision score for each query over the top-10 retrieved results, then averages across all queries.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve. Calculated by varying a similarity threshold to generate precision-recall pairs and integrating the curve.

Input / output format

Input: Query text and a list of candidate passage texts.

Output: Similarity scores for each query-passage pair, used to rank passages.

Scoring recipe

def compute_ndcg_at_k(scores, labels, k=10):
    dcg = sum((2**labels[i] - 1) / math.log2(i + 2) for i in range(min(k, len(scores))))
    ideal = sorted(labels, reverse=True)
    idcg = sum((2**ideal[i] - 1) / math.log2(i + 2) for i in range(min(k, len(ideal))))
    return dcg / idcg if idcg > 0 else 0.0

def compute_mAP_at_k(scores, labels, k=10):
    ranked = sorted(zip(scores, labels), reverse=True)[:k]
    rel_count, p_sum = 0, 0.0
    for i, (_, l) in enumerate(ranked):
        if l == 1:
            rel_count += 1
            p_sum += rel_count / (i + 1)
    return p_sum / max(rel_count, 1)

def compute_auprc(scores, labels):
    thresholds = sorted(set(scores), reverse=True)
    precs, recs = [], []
    for t in thresholds:
        tp = sum(1 for s, l in zip(scores, labels) if s >= t and l == 1)
        fp = sum(1 for s, l in zip(scores, labels) if s >= t and l == 0)
        fn = sum(1 for s, l in zip(scores, labels) if s < t and l == 1)
        precs.append(tp / (tp + fp) if (tp + fp) > 0 else 0.0)
        recs.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return np.trapz(precs, recs)

Common pitfalls

• The custom held-out dataset has exactly one relevant passage per query, making standard MTEB retrieval metrics unsuitable; AUPRC is used instead.
• Hard-label fine-tuning on this small dataset actually degrades performance compared to the benchmark, contrary to expectations that more data/labels always help.
• MTEB includes non-retrieval tasks (e.g., bitext mining), but the evaluation strictly focuses on the retrieval subsets.

Evidence (verbatim from paper)

The evaluation metrics used are mean average precision at 10 (mAP@10), normalized discounted cumulative gain at 10 (nDCG@10), mean reciprocal rank at 10 (mRR@10), and more (Muennighoff et al., [2022]). In this work, we focus on the retrieval subsets. For retrieval tasks, the default metric suggested by MTEB is nDCG@10, while we show both mAP@10 and nDCG@10 metrics for evaluation. ... The metric of the area under precision-recall curve (AUPRC) is considered.

Citation

@misc{lu2024improvingembedding,
  title={Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2408.11868}
}

• arXiv: 2408.11868