Mair Eval

mair-eval

MAIR: A Massive Benchmark for Evaluating Instructed Retrieval — Sun et al. (2024) (arXiv:2410.10127, 2024)

What this evaluates

Evaluates retrieval models' ability to follow complex, task-specific instructions across diverse domains and long-tail tasks. It measures how instruction tuning impacts generalization and performance on heterogeneous query-document relevance tasks compared to non-instruction-tuned baselines.

Datasets

• MAIR — total ?; splits: test (-1); repo https://github.com/sunnweiwei/Mair

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Calculated as the ratio of the DCG@10 of the predicted ranking to the ideal DCG@10. The overall score is the average across all queries, with domain-level averages also reported.

Input / output format

Input: For embedding models: a query paired with a task-specific instruction. For re-rankers: a query, instruction, and a candidate passage (from top-100 retrieved by a first-stage retriever). Passages are truncated to the model's maximum input length.

Output: A ranked list of passages (top-100 for re-rankers, evaluated at top-10).

Scoring recipe

import math
def compute_ndcg_at_10(gold_relevance, predicted_ranking):
    dcg = 0.0
    for i, doc_id in enumerate(predicted_ranking[:10]):
        rel = gold_relevance.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    ideal_rels = sorted(gold_relevance.values(), reverse=True)[:10]
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Non-instruction-tuned models are evaluated under the '+ instruction' setting for reference, which may overstate their actual instruction-following capability.
• Re-ranking models rely on a first-stage retriever (text-embedding-3-small) to fetch the top-100 candidates, so their reported performance conflates retrieval and re-ranking quality.
• Passages are truncated to each model's maximum input length, which can degrade relevance scoring for long documents and affect nDCG@10 scores.

Evidence (verbatim from paper)

Following previous work, we use nDCG@10 as the evaluation metric. The overall score is defined as the average score across all queries. We also report the average nDCG@10 for each of the following domains: Web, Academic, Code, Medical, Legal, and Finance.

Citation

@misc{sun2024mair,
  title={MAIR: A Massive Benchmark for Evaluating Instructed Retrieval},
  author={Sun et al. (2024)},
  year={2024},
  note={arXiv:2410.10127}
}

• arXiv: 2410.10127