Hybridrag Bench Eval

hybridrag-bench-eval

How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge — Lin et al. (2026) (arXiv:2602.10210, 2026)

What this evaluates

Evaluates retrieval-augmented models' ability to perform multi-hop reasoning over hybrid knowledge (unstructured text and knowledge graphs) using time-framed, external scientific literature to prevent parametric memorization.

Datasets

• Arxiv-AI — total 863; splits: test (-1); repo https://github.com/junhongmit/HybridRAG-Bench
• Arxiv-CY — total 966; splits: test (-1); repo https://github.com/junhongmit/HybridRAG-Bench
• Arxiv-BIO — total 1040; splits: test (-1); repo https://github.com/junhongmit/HybridRAG-Bench

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match or token-level F1 depending on question type (entity/value vs. open-ended). Not explicitly defined in the provided section.

Input / output format

Input: A question requiring multi-hop reasoning, provided with retrieved unstructured text chunks and/or knowledge graph paths as context.

Output: A predicted answer (entity, value, or short natural language response) and optionally supporting evidence.

Scoring recipe

def score(predictions, gold, question_type):
    if question_type in ['single-hop', 'multi-hop', 'counterfactual']:
        return 1.0 if predictions.strip() == gold.strip() else 0.0
    else:
        return rouge_f1(predictions, gold)

Common pitfalls

• Questions are explicitly filtered to be answerable only from the provided hybrid context, penalizing models that rely on parametric memorization.
• Counterfactual questions require minimal perturbation of original paths; models may fail to recognize the perturbation or hallucinate unsupported conclusions.
• Open-ended questions require synthesizing explanations from multiple evidence pieces, making them sensitive to retrieval precision and context window limits.

Evidence (verbatim from paper)

Together, these components enable reproducible construction of retrieval-intensive benchmarks for evaluating RAG and KG-RAG methods under controlled knowledge settings.

Citation

@misc{lin2026hybridragbench,
  title={How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2602.10210}
}

• arXiv: 2602.10210