qhjqhj00 trec-dragun-eval
Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment — Zhang et al. (2026) (arXiv:2602.24277, 2026)
What this evaluates
Evaluates assistive RAG systems that support news trustworthiness assessment by generating investigative questions and context-rich reports. Probes the model's ability to identify critical aspects of source bias, motivation, and alternative viewpoints, and to synthesize attributed summaries that help readers evaluate credibility.
Datasets
- TREC DRAGUN 2025 Track — total 30; splits: test (30); repo https://github.com/trec-dragun/resources
Metrics
Kendall's τ(primary) — range: [-1, 1]- Measures the rank correlation between automated rubric-based scores and human assessor judgments. It counts concordant and discordant pairs between the predicted and gold rankings to produce a coefficient ranging from -1 to 1.
Input / output format
Input: A target news article (topic) from the MS MARCO V2.1 Document Corpus. For Task 1, the article is provided to generate investigative questions. For Task 2, the article is provided as context to generate a trustworthiness report.
Output: Task 1: A ranked list of up to 10 critical questions, each ≤300 characters, non-compound, and specific to the article. Task 2: A ~250-word report where each sentence contains at most three cited segment IDs from the MS MARCO V2.1 Segmented Corpus.
Scoring recipe
def compute_kendall_tau(predictions, gold, rubrics):
auto_scores = []
for pred, topic_id in zip(predictions, rubrics.keys()):
# AutoJudge matches prediction against importance-weighted rubric
score = compute_rubric_match(pred, rubrics[topic_id])
auto_scores.append(score)
# Compare automated scores with human judgments
tau = kendall_correlation(auto_scores, gold)
return tau
Common pitfalls
- Questions must be strictly ranked by importance and cannot be compound or overly general.
- Report sentences are limited to a maximum of three citation segment IDs per sentence.
- Generated questions must not exceed 300 characters in length.
- Reports must be approximately 250 words and explicitly attribute claims to retrieved segments.
Evidence (verbatim from paper)
It leverages human-created importance-weighted rubrics from TREC assessors—developed to identify key investigative questions and evidence for 30 news articles—to automatically score generated reports and questions. AutoJudge achieves strong correlation (Kendall’s τ = 0.872 for report generation, 0.678 for question generation) with human judgments, enabling scalable, reusable evaluation of RAG systems in lateral reading workflows where readers need to assess credibility through multi-source context and investigative guidance.
Citation
@misc{zhang2026resources,
title={Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment},
author={Zhang et al. (2026)},
year={2026},
note={arXiv:2602.24277}
}
- arXiv: 2602.24277