Bright Eval

bright-eval

BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval — Hongjin Su et al. (2024) (arXiv:2407.12883, 2024)

What this evaluates

Evaluates a model's ability to perform reasoning-intensive text retrieval by matching complex, domain-diverse queries to relevant documents. It probes deep logical and conceptual alignment between queries and documents, going beyond simple keyword or semantic matching.

Datasets

• BRIGHT — total ?; splits: test (-1); repo https://github.com/xlang-ai/BRIGHT

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It computes the discounted gain of relevant documents in a ranked list, normalizing by the ideal DCG to yield a score between 0 and 1.

Input / output format

Input: A query (original user query or LLM-generated reasoning trace) and a document corpus to search over.

Output: A ranked list of retrieved documents (top-10).

Scoring recipe

def ndcg_at_10(relevance_scores, k=10):
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• LLM-based QA evaluation may not accurately reflect retrieval quality because the generator might fail to integrate retrieved documents, or the evaluator might struggle to compare open-ended answers.
• Trained dense retrievers often underperform on out-of-distribution reasoning queries, while BM25 adapts better to novel query formulations.

Evidence (verbatim from paper)

Following prior work (Thakur et al., 2021; Bajaj et al., 2018; Voorhees & Tice, 2000), we use nDCG@10 as the main metric. Results in Table 2 show that BRIGHT is very challenging, with the best model achieving only $24.3\mathrm{nDCG}@10$.

Citation

@misc{su2024bright,
  title={BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval},
  author={Hongjin Su et al. (2024)},
  year={2024},
  note={arXiv:2407.12883}
}

• arXiv: 2407.12883