Trec2025 RAG Eval

trec2025-rag-eval

Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track — Upadhyay et al. (2026) (arXiv:2603.09891, 2026)

What this evaluates

Probes retrieval-augmented generation systems on complex, narrative-driven queries by decomposing information needs into sub-narratives. It evaluates document relevance based on sub-narrative coverage, measures response quality via strict vital recall of fully supported information nuggets, and assesses sentence-level factual grounding against cited documents.

Datasets

• MS MARCO V2.1 — total ?; splits: test (-1)

Metrics

• strict_vital_recall (primary) — range: [0, 1]
  • Recall calculated over all vital nuggets extracted from documents with relevance ≥ 1. A nugget is counted as retrieved only if the system response fully supports it.
• relevance_score — range: [0, 4]
  • Integer scale 0-4 based on the number of sub-narratives a passage addresses in detail. Downgraded by 1 or 2 for irrelevant extra information.
• sub_narrative_coverage — range: [0, 1]
  • Binary metric per sub-narrative; considered covered if at least one mapped vital nugget is fully supported by the response.

Input / output format

Input: Per instance: a first-person narrative, a list of decomposed sub-narratives, and a retrieved passage. For response evaluation: the system-generated answer and the list of vital nuggets with their corresponding sub-narrative mappings.

Output: For relevance: integer score 0-4 formatted as '##final score: X'. For response evaluation: support labels (Full Support, Partial Support, No Support) per nugget/sentence, and a binary coverage flag per sub-narrative.

Scoring recipe

def calc_relevance(sub_narratives, passage):
    covered = count_detailed_answers(sub_narratives, passage)
    if covered >= 4: score = 4
    elif covered >= 2: score = 3
    elif covered == 1: score = 2
    elif has_related_but_no_answer(passage): score = 1
    else: score = 0
    score -= count_irrelevant_extra_info(passage)
    return max(0, min(4, score))

def calc_vital_recall(vital_nuggets, response):
    supported = sum(1 for n in vital_nuggets if is_fully_supported(n, response))
    return supported / len(vital_nuggets) if vital_nuggets else 0

Common pitfalls

• Treating vague mentions or partial coverage as full support for a sub-narrative or vital nugget.
• Ignoring the mandatory downgrade penalty when a passage contains irrelevant extra information.
• Evaluating factual support at the paragraph level instead of the required sentence-level granularity.

Evidence (verbatim from paper)

For the assignment, strict vital recall is used, as it is found to be best suited for response evaluation in the TREC RAG 2024 Track. In particular, this metric calculates recall over all vital nuggets that are fully supported by the response.

Citation

@misc{upadhyay2026trec2025rag,
  title={Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track},
  author={Upadhyay et al. (2026)},
  year={2026},
  note={arXiv:2603.09891}
}

• arXiv: 2603.09891