RAG Coverage Eval

rag-coverage-eval

Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage — Samuel et al. (2026) (arXiv:2603.08819, 2026)

What this evaluates

This evaluation probes the relationship between retrieval effectiveness and downstream information coverage in RAG systems. It measures how well retrieval models capture required information nuggets and how accurately generated responses cover these nuggets with proper citations.

Datasets

• NeuCLIR24 — total 19; splits: test (19)
• RAG24 — total 55; splits: test (55)
• WikiVideo — total 57; splits: test (57)

Metrics

• Nugget Coverage (primary) — range: [0, 1]
  • Proportion of grounded nuggets covered in the generated response. A nugget is grounded only if accompanied by a citation to a document containing it.
• α-nDCG — range: [0, 1]
  • Discounts gain by document rank and reduces gain when a nugget is already covered by an earlier document. Computed as DCG/IDCG with rank cutoff 20 (or 10 for WikiVideo).
• Subtopic Recall (StRecall) — range: [0, 1]
  • A set measure without ranking penalty that calculates the fraction of total nuggets covered by the retrieved documents.
• Information Recall (InfoR) — range: [0, 1]
  • Directly assesses whether a nugget is covered in the response, without requiring a citation.

Input / output format

Input: Query (problem statement or question) and a document/video collection for retrieval; query and retrieved documents/videos for RAG generation.

Output: Ranked list of documents/videos for retrieval; generated report/response with citations for RAG.

Scoring recipe

def compute_nugget_coverage(predictions, gold_nuggets):
    covered = 0
    for nugget in gold_nuggets:
        if nugget in predictions and predictions[nugget].has_citation:
            covered += 1
    return covered / len(gold_nuggets)

def compute_alpha_ndcg(ranked_docs, gold_nuggets, cutoff=20):
    covered = set()
    dcg = 0.0
    for rank, doc in enumerate(ranked_docs[:cutoff], 1):
        new = set(doc.nuggets) - covered
        covered.update(new)
        dcg += len(new) / math.log2(rank + 1)
    idcg = len(gold_nuggets) / math.log2(cutoff + 1)
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Confusing nugget-based nDCG with relevance-based nDCG; the latter does not require documents to contain nuggets to be relevant.
• Overlooking the citation requirement for Nugget Coverage; only cited nuggets count as covered, unlike InfoR which counts any covered nugget.
• Using rank correlation instead of Pearson correlation when assessing the relationship between retrieval and RAG metrics, as the paper explicitly prefers Pearson to capture value relationships rather than system rankings.

Evidence (verbatim from paper)

We report three coverage metrics, α-nDCG, nDCG using the nugget-based qrels, and Subtopic Recall (StRecall). ... We evaluate generated responses using Nugget Coverage, which is the proportion of the grounded nuggets covered in the generated response. A nugget is grounded if it is accompanied by a citation to a document containing the nugget.

Citation

@misc{samuel2026beyond,
  title={Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage},
  author={Samuel et al. (2026)},
  year={2026},
  note={arXiv:2603.08819}
}

• arXiv: 2603.08819