Dragon RAG Eval

dragon-rag-eval

DRAGON: Dynamic RAG Benchmark On News — Chernogorskii et al. (2025) (arXiv:2507.05713, 2025)

What this evaluates

Evaluates retrieval and end-to-end performance of RAG systems on a dynamic, daily-updating news corpus. It probes a model's ability to accurately retrieve relevant document chunks and generate factually consistent responses to knowledge-graph-derived queries.

Datasets

• Public Texts — total ?; splits: test (-1); repo https://github.com/RussianNLP/DRAGON

Metrics

• Hit Rate — range: [0, 1]
  • Fraction of queries where the ground-truth relevant document appears in the top-5 retrieved chunks.
• Recall — range: [0, 1]
  • Fraction of ground-truth relevant documents successfully retrieved among the top-5 chunks.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5, measuring the ranking quality of retrieved chunks.
• ROUGE-L (primary) — range: [0, 1]
  • Longest common subsequence F1 score between the generated response and the reference answer.
• Substring Matching — range: [0, 1]
  • Exact substring match score between generated and reference text.
• Judge Score — range: [0, 1]
  • LLM-as-a-judge evaluation score for response quality and factual alignment (see Appx. H.4).

Input / output format

Input: User query paired with the top-5 retrieved document chunks, formatted into a system prompt template.

Output: Generated natural language response to the query.

Scoring recipe

def score(predictions, golds, retrieved_docs):
    hit_rate = sum(1 for q in queries if golds[q] in retrieved_docs[:5]) / len(queries)
    recall = sum(len(set(golds[q]) & set(retrieved_docs[:5])) / len(golds[q]) for q in queries) / len(queries)
    ndcg = compute_ndcg_at_k(retrieved_docs, golds, k=5)
    rouge_l = compute_rouge_l(predictions, golds)
    sm = compute_substring_match(predictions, golds)
    judge = compute_llm_judge_score(predictions, golds)
    return {'Hit Rate': hit_rate, 'Recall': recall, 'NDCG': ndcg, 'ROUGE-L': rouge_l, 'Substring Matching': sm, 'Judge Score': judge}

Common pitfalls

• Retrieval is strictly limited to the top-5 chunks; using more or fewer will invalidate Hit Rate/Recall/NDCG.
• Prompt context must be truncated if it exceeds the model's max context length, which can drop retrieved chunks and hurt retrieval metrics.
• Chunking must use exactly 500-character length with 100-character overlap; different splitting alters vectorization and retrieval alignment.

Evidence (verbatim from paper)

The performance of retrieval is measured with the 3 main metrics: Hit Rate, Recall, and NDCG. End-to-end RAG-system evaluation is performed via ROUGE-L, Substring Matching (SM), and Judge Score. See Appx.[H.4] for their description.

Citation

@misc{chernogorskii2025dragon,
  title={DRAGON: Dynamic RAG Benchmark On News},
  author={Chernogorskii et al. (2025)},
  year={2025},
  note={arXiv:2507.05713}
}

• arXiv: 2507.05713