RAG Reasoning Eval

rag-reasoning-eval

Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization — Wei et al. (2025) (arXiv:2504.14858, 2025)

What this evaluates

Evaluates retrieval-augmented reasoning systems on their ability to iteratively refine answers using a critique language model. It probes robustness to noisy retrieval, out-of-distribution generalization, and the effectiveness of contrastive critique synthesis over standard self-refinement baselines.

Datasets

• PopQA — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• NaturalQuestions — total ?; splits: test (-1)
• 2WikiMultihopQA — total ?; splits: test (-1)
• ASQA — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• SQuAD — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Binary score (1 if the final generated answer exactly matches the ground-truth answer, 0 otherwise), averaged over all instances in the dataset.
• str-em — range: [0, 1]
  • Official string exact-match metric used for ASQA. Scores 1 if the prediction matches the reference string exactly, 0 otherwise.

Input / output format

Input: Question text and top-5 retrieved passages.

Output: Final generated answer string.

Scoring recipe

def compute_metrics(predictions, golds, dataset_name):
    scores = []
    for pred, gold in zip(predictions, golds):
        if dataset_name == 'ASQA':
            scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
        else:
            scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
    return sum(scores) / len(scores) if scores else 0.0

Common pitfalls

• ASQA uses string exact match (str-em) instead of standard accuracy, which can penalize valid paraphrases or multi-sentence answers.
• The 'fixed' variant uses exactly 1 refinement iteration, while 'auto' dynamically terminates, making direct comparison of iteration counts misleading.
• Retrieval is strictly limited to Top-5 passages for all experiments, which may not reflect full RAG pipeline performance or optimal retrieval settings.

Evidence (verbatim from paper)

Following previous work[[51]], we adopt the official correctness metric (str-em) for ASQA[[45]], and use accuracy for the other tasks, which measures whether the final generations of the model align with the ground-truth[[41], [52]].

Citation

@misc{wei2025ragreasonalignment,
  title={Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2504.14858}
}

• arXiv: 2504.14858