Ragsearch Eval

ragsearch-eval

Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems — Fan et al. (2026) (arXiv:2604.09666, 2026)

What this evaluates

Evaluates dense RAG and GraphRAG retrieval backends when integrated into agentic search systems. It probes the agent's ability to dynamically retrieve, reason, and answer general and multi-hop QA queries under both training-free prompting and reinforcement learning paradigms.

Datasets

• NQ — total ?; splits: test (-1)
• PopQA — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• 2Wiki — total ?; splits: test (-1)
• Musique — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer exactly matches the gold answer string, 0 otherwise. Reported as the overall percentage across the dataset.

Input / output format

Input: A natural language query q and a retrieval backend B (dense RAG or GraphRAG). The agent interacts iteratively, receiving retrieved text chunks or subgraphs wrapped in tags.

Output: A final answer enclosed within and tags, following an interleaved sequence of and steps.

Scoring recipe

def compute_em(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            correct += 1
    return correct / len(golds)

Common pitfalls

• Confusing the agentic control policy with the retrieval backend; the benchmark keeps the agent protocol identical across Dense RAG and GraphRAG to isolate retrieval infrastructure effects.
• Failing to account for the multi-step agentic loop; single-shot baselines are evaluated separately from training-free and RL-based multi-round agents.
• GraphRAG backends require offline graph construction, but the benchmark metric only measures runtime answer correctness, not preprocessing time or cost.

Evidence (verbatim from paper)

In RAGSearch, rewards are defined at the trajectory level and focus on task correctness and output validity. Specifically, we combine (i) an outcome-based reward that measures answer correctness (e.g., exact match or task-specific accuracy), and (ii) a format-based reward that encourages the agent to follow the expected interaction and answer format.

Citation

@misc{fan2026ragsearch,
  title={Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems},
  author={Fan et al. (2026)},
  year={2026},
  note={arXiv:2604.09666}
}

• arXiv: 2604.09666