Trec2024 RAG Nugget Eval

trec2024-rag-nugget-eval

Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework — Pradeep et al. (2024) (arXiv:2411.09607, 2024)

What this evaluates

Evaluates the factual accuracy and content grounding of RAG-generated answers by checking for the presence of key factual claims (nuggets) extracted from source documents. It also measures answer length to assess the trade-off between conciseness and completeness in system outputs.

Datasets

• TREC 2024 RAG Track — total 301; splits: test (21)

Metrics

• V_strict (primary) — range: [0, 1]
  • Measures the presence of vital nuggets (key factual claims) in the generated answer relative to the reference documents. Calculated as the proportion of vital nuggets successfully identified in the output.
• L — range: other
  • Answer length measured as the total number of standard whitespace-separated words in the generated response.
• Kendall's tau — range: [-1, 1]
  • Rank correlation coefficient used to measure agreement between manual and automatic evaluation scores across runs or topics.

Input / output format

Input: Query/topic and a set of reference documents (for RAG track); query only (for AG track).

Output: Generated answer text (guidelines cap at 400 words, though some submissions were shorter).

Scoring recipe

# Extract vital nuggets from reference documents (via AutoNuggetizer or manual curation)
vital_nuggets = get_vital_nuggets(reference_docs)
# Check presence of each nugget in the generated answer
found = [n for n in vital_nuggets if is_factual_claim_present(n, generated_answer)]
# Compute strict nugget score
V_strict = len(found) / len(vital_nuggets) if vital_nuggets else 0
# Compute answer length
L = len(generated_answer.split())

Common pitfalls

• Confusing run-level rank correlation with topic-level correlation; the paper notes Kendall's tau is 0.783 at run level but drops to 0.324 when treating each topic/run combination as an independent observation.
• Assuming longer answers inherently yield higher quality; the evaluation reveals a Pareto-optimal trade-off where answer length and factual coverage vary independently.
• Overlooking the difference between evaluation conditions; scores differ significantly between 'AutoNuggets+Edits/ManualAssign' and 'AutoNuggets/AutoAssign' protocols.

Evidence (verbatim from paper)

Our $V_{ extrm{strict}}$ score captures the presence of vital nuggets, but this needs to be balanced by the length of the answer.

Citation

@misc{pradeep2024nuggeteval,
  title={Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework},
  author={Pradeep et al. (2024)},
  year={2024},
  note={arXiv:2411.09607}
}

• arXiv: 2411.09607