Birco Eval

birco-eval

BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives — Wang et al. (2024) (arXiv:2402.14151, 2024)

What this evaluates

This benchmark evaluates LLM-based information retrieval systems on complex, multi-faceted query objectives that go beyond simple lexical or semantic similarity. It probes whether models can correctly rank documents based on structured tasks like refuting claims, measuring drug effects, or identifying specific book details, often requiring explicit task understanding rather than just passage matching.

Datasets

• DORIS-MAE — total ?; splits: test (-1)
• ArguAna — total ?; splits: test (-1)
• WhatsThatBook — total ?; splits: test (-1)
• Clinical-Trial — total ?; splits: test (-1)
• RELIC — total ?; splits: test (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores for the top 10 retrieved documents, normalized by the ideal DCG for that query.
• Recall@5 — range: [0, 1]
  • The proportion of relevant documents found within the top 5 retrieved results, relative to the total number of relevant documents in the candidate set.

Input / output format

Input: A paragraph-length query and a candidate set of documents (passages) to be ranked or scored.

Output: A ranked list of documents or relevance scores for each candidate document.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of doc IDs ranked by model
    # gold: set of relevant doc IDs
    # nDCG@10
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(predictions[:10]) if doc in gold)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold), 10)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    # Recall@5
    recall = len(set(predictions[:5]) & gold) / len(gold) if gold else 0.0
    return ndcg, recall

Common pitfalls

• Hard negatives are explicitly constructed to be highly similar to ground truth, significantly increasing retrieval difficulty and lowering recall.
• Tasks like ArguAna and RELIC have non-standard objectives; models without explicit task descriptions perform poorly due to ambiguity in relevance criteria.
• Chain-of-thought prompting does not consistently improve performance and may slightly decrease it on aggregate across tasks.

Evidence (verbatim from paper)

Table 5: nDCG@10 and Recall@5 for the benchmark datasets. Bold indicates no statistically significant difference from the highest numerical value, which is indicated in red. Refer to Appendix Table 7 for standard errors. The notation +O indicates task objective awareness. The consistently low recall on DORIS-MAE and Clinical-Trial corresponds to their low maximum possible recall.

Citation

@misc{wang2024birco,
  title={BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2402.14151}
}

• arXiv: 2402.14151