Trec RAG Support Eval

trec-rag-support-eval

Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges — Thakur et al. (2025) (arXiv:2504.15205, 2025)

What this evaluates

Evaluates the ability of LLM judges versus human annotators to assess sentence-level grounding (support) in RAG-generated answers. It measures how well models cite relevant passages and whether the cited text actually supports the generated claims.

Datasets

• TREC 2024 RAG Track — total ?; splits: test (-1)

Metrics

• weighted precision (primary) — range: [0, 1]
  • Average precision score across topics and runs, computed using weighted support labels (full, partial, no support) to reflect the degree of grounding in the cited text.
• Kendall’s τ — range: [-1, 1]
  • Rank correlation coefficient measuring the agreement between run-level scores of GPT-4o and human judges across topics.
• Cohen’s κ — range: [-1, 1]
  • Inter-annotator agreement score measuring consistency between independent judges on cases where primary human and LLM judgments disagree.

Input / output format

Input: Query, RAG-generated answer, and cited sentence-passage pairs. Judges assess whether the cited text supports the answer.

Output: Categorical support label per instance: 'full support', 'partial support', or 'no support'.

Scoring recipe

label_weights = {'full support': 1.0, 'partial support': 0.5, 'no support': 0.0}
weighted_scores = [label_weights[pred] for pred in predictions]
avg_precision = sum(weighted_scores) / len(predictions)
# Run-level correlation
kendall_tau = scipy.stats.kendalltau(human_scores, llm_scores).correlation
# Inter-annotator agreement on disagreements
cohen_kappa = sklearn.metrics.cohen_kappa_score(human_labels, llm_labels)

Common pitfalls

• Human annotators tend to be more conservative, preferring 'no support' and yielding lower weighted precision/recall scores than GPT-4o.
• Agreement rates vary drastically between conditions: 56% perfect agreement in from-scratch vs. 72.1% in post-editing conditions.
• GPT-4o systematically assigns higher support labels than humans, particularly favoring 'partial support' over 'no support'.

Evidence (verbatim from paper)

We sort the runs in terms of the average weighted precision score in descending order. Run-level scores (denoted by $ imes$) are strongly correlated (all above 0.79 Kendall’s $ au$) between GPT-4o and human annotations. For 56% (13.7% + 11.9% + 30.4%), GPT-4o and the human judge perfectly agreed on their support judgment on 22 topics.

Citation

@misc{thakur2025supportevaluation,
  title={Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges},
  author={Thakur et al. (2025)},
  year={2025},
  note={arXiv:2504.15205}
}

• arXiv: 2504.15205