Beir Eval

beir-eval

Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard — Kamalloo et al. (2023) (arXiv:2306.07471, 2023)

What this evaluates

Evaluates zero-shot information retrieval capabilities across 18 diverse domains and query types. It probes a model's ability to retrieve relevant documents without domain-specific fine-tuning, highlighting performance variations due to domain shifts, query length, and lexical versus semantic matching.

Datasets

• BEIR — total ?; splits: test (-1); HF beir/benchmark; repo https://github.com/beir-cellar/beir

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores for the top 10 retrieved documents, discounted logarithmically by position, and normalized by the ideal DCG for that query.
• recall@100 — range: [0, 1]
  • Recall at rank 100. Calculates the fraction of all relevant documents in the corpus that appear within the top 100 retrieved results.

Input / output format

Input: Query text and a target corpus of documents. Models receive queries and must rank documents from the corpus.

Output: Ranked list of document IDs or passages. Results are typically reported for top-10 and top-100 ranks.

Scoring recipe

def compute_metrics(predictions, gold):
    ndcg_vals, rec_vals = [], []
    for qid, ranked in predictions.items():
        rel_set = gold[qid]
        dcg = sum(1.0 / log2(i + 2) for i, doc in enumerate(ranked[:10]) if doc in rel_set)
        idcg = sum(1.0 / log2(i + 2) for i in range(min(10, len(rel_set))))
        ndcg_vals.append(dcg / idcg if idcg > 0 else 0.0)
        hits = sum(1 for doc in ranked[:100] if doc in rel_set)
        rec_vals.append(hits / len(rel_set) if rel_set else 0.0)
    return {'nDCG@10': mean(ndcg_vals), 'recall@100': mean(rec_vals)}

Common pitfalls

• Macro-averaging scores across all 18 BEIR datasets can obscure severe performance drops on specific domains (e.g., biomedical or short-query tasks).
• Relevance judgments in some datasets (e.g., Toulouse-2020) are shallow and biased toward lexical matches, unfairly penalizing semantic retrieval models.
• Zero-shot evaluation means no domain-specific fine-tuning is allowed, so results heavily reflect pre-training data overlap rather than true generalization.

Evidence (verbatim from paper)

The effectiveness of the five models presented in the previous section is shown in Table 2, with nDCG@10 in the left group of columns and recall@100 in the right group of columns. Each row corresponds to one of the BEIR datasets, and the rows are ordered in the same manner as Thakur et al. [34].

Citation

@misc{kamalloo2023beir,
  title={Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard},
  author={Kamalloo et al. (2023)},
  year={2023},
  note={arXiv:2306.07471}
}

• arXiv: 2306.07471