Graphrag Bench Eval

graphrag-bench-eval

When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation — Xiang et al. (2025) (arXiv:2506.05690, 2025)

What this evaluates

Evaluates Graph Retrieval-Augmented Generation (GraphRAG) frameworks against vanilla RAG across fact retrieval, complex reasoning, contextual summarization, and creative generation tasks. It measures generation quality, retrieval effectiveness, graph structural complexity, and computational efficiency to determine when graph-based retrieval provides measurable benefits over dense vector retrieval.

Datasets

• Novel Dataset — total ?; splits: test (-1); repo https://github.com/GraphRAG-Bench/GraphRAG-Benchmark
• Medical Dataset — total ?; splits: test (-1); repo https://github.com/GraphRAG-Bench/GraphRAG-Benchmark

Metrics

• ROUGE-L — range: 0-1
  • Standard ROUGE-L F1 score comparing the generated answer to the gold reference answer.
• Accuracy (ACC) — range: percent
  • Proportion of questions where the generated answer exactly matches or is judged correct against the gold answer.
• Evidence Coverage (Cov) — range: percent
  • LLM-judged score measuring how comprehensively the generated answer covers the provided gold evidence.
• Faithfulness (FS) — range: percent
  • LLM-judged score assessing the factual consistency of the generated answer against the gold evidence.
• Evidence Recall (primary) — range: percent
  • Measures how completely the retrieved context chunks cover the gold evidence passages.
• Context Relevance — range: percent
  • LLM-judged score measuring the semantic alignment between the retrieved content and the input query.

Input / output format

Input: Query/question paired with a document corpus. Questions are categorized into four types: fact retrieval, complex reasoning, contextual summarization, and creative generation.

Output: Generated answer text and retrieved context chunks/documents.

Scoring recipe

def compute_metrics(predictions, golds, retrieved_contexts, queries, gold_evidence):
    # Generation metrics
    rouge_l = compute_rouge_l(golds, predictions)
    acc = mean([1.0 if p == g else 0.0 for p, g in zip(predictions, golds)])
    cov = llm_judge_gpt4o_mini('coverage', gold_evidence, predictions)
    fs = llm_judge_gpt4o_mini('faithfulness', gold_evidence, predictions)
    # Retrieval metrics
    recall = compute_recall(retrieved_contexts, gold_evidence)
    relevance = llm_judge_gpt4o_mini('relevance', queries, retrieved_contexts)
    return {'ROUGE-L': rouge_l, 'ACC': acc, 'Cov': cov, 'FS': fs, 'Recall': recall, 'Relevance': relevance}

Common pitfalls

• Relies heavily on GPT-4o-mini as an LLM-as-a-judge for coverage, faithfulness, recall, and relevance, which may introduce bias or inconsistency compared to exact-match metrics.
• GraphRAG's token overhead and prompt inflation can degrade context relevance on simple tasks, making it appear worse than vanilla RAG despite higher retrieval recall.
• Evaluation splits and exact dataset sizes are not disclosed, making reproducibility and cross-study comparison difficult.

Evidence (verbatim from paper)

For Type 1 (retrieval) and Type 2 (reasoning) questions, we assess answer quality with ROUGE scores and accuracy. For Type 3 (summarization) questions, we introduce evidence coverage to measure the comprehensiveness of the generated answers. For Type 4 (creative generation) questions, we use faithfulness to assess factual consistency. To quantitatively compare the retrieval effectiveness of the two paradigms, we adopt two complementary metrics: Evidence Recall, which measures how completely the retrieved context covers the gold evidence, and Context Relevance, which measures the semantic alignment between the retrieved content and the input query.

Citation

@misc{xiang2025whentousegraphs,
  title={When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation},
  author={Xiang et al. (2025)},
  year={2025},
  note={arXiv:2506.05690}
}

• arXiv: 2506.05690