Retrieval Robustness Eval

retrieval-robustness-eval

Evaluating the Retrieval Robustness of Large Language Models — Cao et al. (2025) (arXiv:2505.21870, 2025)

What this evaluates

This evaluation probes how consistently large language models maintain or improve their answer quality when provided with retrieved context, specifically measuring resilience to variations in retrieval size, document order, and the risk of performance degradation compared to non-retrieval baselines.

Datasets

• Wikipedia QA benchmark — total 1500; splits: test (1500)

Metrics

• No-Degradation Rate (NDR) (primary) — range: percent
  • The percentage of questions where the model's answer with retrieval is at least as good as its answer without retrieval. Calculated as the proportion of samples where score(RAG) >= score(Non-RAG).
• Retrieval Size Robustness (RSR) — range: percent
  • The stability of task performance as the number of retrieved documents (k) increases. Measured by comparing actual performance across k values against an oracle setup that selects the best answer among all k values.
• Retrieval Order Robustness (ROR) — range: percent
  • The stability of task performance when the order of retrieved documents is permuted. Measured by comparing actual performance across different orders against an oracle setup that selects the best response among all order permutations.

Input / output format

Input: A natural language question, a set of k retrieved documents from Wikipedia (varying in k and order), and a system prompt (e.g., vanilla RAG prompt).

Output: A generated text response answering the question.

Scoring recipe

def compute_metrics(questions, model, evaluator):
    ndr_count = 0
    rsr_gaps, ror_gaps = [], []
    for q in questions:
        ans_nonrag = model(q)
        score_nonrag = evaluator(q, ans_nonrag)
        scores_by_k = {}
        scores_by_o = {}
        for k in [1, 10, ..., 75]:
            for order in permutations:
                ans_rag = model(q, retrieved_docs[k, order])
                score_rag = evaluator(q, ans_rag)
                scores_by_k.setdefault(k, []).append(score_rag)
                scores_by_o.setdefault(order, []).append(score_rag)
        if any(s >= score_nonrag for s in scores_by_k.values()):
            ndr_count += 1
        oracle_k = max(max(v) for v in scores_by_k.values())
        oracle_o = max(max(v) for v in scores_by_o.values())
        rsr_gaps.append(oracle_k - mean(scores_by_k.values()))
        ror_gaps.append(oracle_o - mean(scores_by_o.values()))
    return ndr_count/len(questions), mean(rsr_gaps), mean(ror_gaps)

Common pitfalls

• Confusing overall task performance with retrieval robustness; larger models often have higher absolute performance but lower NDR due to stronger parametric knowledge.
• Assuming robustness metrics imply uniform sample-level performance; models frequently trade off performance across individual examples when changing k or order.
• Overlooking that 'perfect' oracle setups (perfect NDR/RSR/ROR) are hypothetical baselines used to quantify the gap, not actual evaluation conditions.

Evidence (verbatim from paper)

This oracle setup assumes perfect NDR, meaning the models consistently generate responses at least as good as those produced without retrieval.

Citation

@misc{cao2025evaluating,
  title={Evaluating the Retrieval Robustness of Large Language Models},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2505.21870}
}

• arXiv: 2505.21870