Rar B Eval

rar-b-eval

RAR-b: Reasoning as Retrieval Benchmark — Xiao et al. (2024) (arXiv:2404.06347, 2024)

What this evaluates

Evaluates whether dense retrievers and re-rankers can semantically encode and retrieve correct answers to reasoning problems across diverse tasks. It probes the retriever-LLM behavioral gap by testing performance with and without task instructions, and compares full-dataset retrieval against multiple-choice retrieval settings.

Datasets

• RAR-b — total ?; splits: test (-1); repo https://github.com/gowitheflow-1998/RAR-b

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computed per task and then averaged using the geometric mean across tasks: G = (∏_{i=1}^{n} x_i)^{1/n}, where n is the number of tasks and x_i is the performance on task i.

Input / output format

Input: A reasoning question or query, optionally prepended with a task instruction. The model retrieves from a candidate pool consisting of either all available answers/documents or a multiple-choice set.

Output: A ranked list of candidate answers/documents. Performance is evaluated based on the rank of the ground-truth answer within the top-10 results.

Scoring recipe

def compute_ndcg_at_10(retrieved, gold, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(retrieved[:k]) if doc == gold)
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0

def evaluate_rar_b(predictions, golds, tasks):
    task_scores = []
    for task in tasks:
        scores = [compute_ndcg_at_10(p, g) for p, g in zip(predictions[task], golds[task])]
        task_scores.append(sum(scores) / len(scores))
    return math.exp(sum(math.log(s) for s in task_scores) / len(task_scores))

Common pitfalls

• Using arithmetic mean instead of geometric mean to average performance across tasks, which biases results toward easier tasks due to vastly different performance scales.
• Ignoring the instruction condition: models not trained for instruction-following degrade when instructions are prepended, masking true retrieval capability.
• Full-dataset retrieval can be gamed by simple entity/keyword matching (e.g., in Winogrande), whereas the Multiple-choice Retrieval (MCR) setting is required to reveal nuanced understanding.

Evidence (verbatim from paper)

Table[2] presents the results for nDCG@10 performance. Because of the different scales of nDCG@10 across tasks due to different task difficulties and corpus sizes, we take the geometric mean across tasks to represent each model’s average performance, which is given by $G=\left(\prod_{i=1}^{n}x_{i}\right)^{\frac{1}{n}}$, where $n$ is the number of tasks and $x_{i}$ represent the performance of each task.

Citation

@misc{xiao2024rarb,
  title={RAR-b: Reasoning as Retrieval Benchmark},
  author={Xiao et al. (2024)},
  year={2024},
  note={arXiv:2404.06347}
}

• arXiv: 2404.06347