Reranker Benchmark Eval

reranker-benchmark-eval

From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars — Kornilov et al. (2024) (arXiv:2411.15577, 2024)

What this evaluates

Evaluates the ability of embedding and reranking models to retrieve and rank relevant paragraphs from descriptive linguistic grammars based on typological feature queries, specifically testing their capacity to filter noisy or partially relevant context.

Datasets

• The Benchmark for Rerankers — total 700; splits: test (700); repo https://github.com/al-the-eigenvalue/RAG-on-grammars

Metrics

• NDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k, computed over a graded relevance scale of 0–5 (where 0 is irrelevant, 5 is unambiguously determinable). It normalizes the DCG by the ideal DCG (IDCG) for the given query.

Input / output format

Input: A query string (either a specific term like 'Dominant word order (Order of Subject, Object, and Verb)' or a Wikipedia summary) paired with an instruction prompt, used to retrieve and rank a collection of 700 annotated paragraphs from 14 descriptive grammars.

Output: A ranked list of paragraphs (top-k) returned by the reranker/embedding model.

Scoring recipe

def ndcg_at_k(predictions, gold, k=20):
    dcg = 0.0
    for i, (doc_id, _) in enumerate(predictions[:k]):
        rel = gold.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    ideal_rels = sorted(gold.values(), reverse=True)[:k]
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Using a binary relevance scale (0/1) instead of the specified 0–5 graded scale, which invalidates the metric's advantage over Recall/MAP.
• Assuming high MTEB leaderboard rankings predict strong performance on this domain; the paper reports a strong negative correlation (Spearman’s ρ = -0.8571).
• Ignoring the impact of instruction tuning and query formulation, which cause significant score variations across models.

Evidence (verbatim from paper)

As the metric for evaluating the rerankers, we chose NDCG@k (Normalized Discounted Cumulative Gain at k) (Järvelin and Kekäläinen, [2002]) over other metrics commonly used for evaluation of information retrieval systems: Recall@k, Mean Average Precision@k (MAP@k), and Mean Reciprocal Rank (MRR), since NDCG@k is the only metric among them that can take into account a scale of more than two relevant ranks: our scale contains six different categories of relevance (0-5) instead of a binary “1 = relevant, 0 = not relevant” distinction.

Citation

@misc{kornilov2024mteb,
  title={From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars},
  author={Kornilov et al. (2024)},
  year={2024},
  note={arXiv:2411.15577}
}

• arXiv: 2411.15577