Multihop RAG Eval

multihop-rag-eval

MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries — Tang et al. (2024) (arXiv:2401.15391, 2024)

What this evaluates

Evaluates retrieval-augmented generation (RAG) systems on multi-hop queries that require retrieving and reasoning across multiple evidence sources. It probes both the retrieval component's ability to find relevant text chunks and the generation component's ability to synthesize accurate answers from retrieved or ground-truth evidence.

Datasets

• MultiHop-RAG — total 2556; splits: test (2556)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly generated answers compared to ground-truth answers. Calculated as correct predictions divided by total non-null queries.
• Hits@10 — range: [0, 1]
  • Binary indicator of whether at least one ground-truth evidence chunk appears in the top-10 retrieved chunks.
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank over top-10 results; averages 1/rank of the first relevant chunk across queries.
• MAP@10 — range: [0, 1]
  • Mean Average Precision over top-10 results; averages precision at each relevant chunk position in the top-10 list.
• Hits@4 — range: [0, 1]
  • Binary indicator of whether at least one ground-truth evidence chunk appears in the top-4 retrieved chunks.

Input / output format

Input: Query string (multi-hop question). For retrieval: query only. For generation: query + retrieved text chunks (or ground-truth evidence).

Output: Retrieved top-K text chunks (retrieval task) or generated natural language answer/response (generation task).

Scoring recipe

# Retrieval scoring
def score_retrieval(pred_chunks, gold_chunks, k=10):
    top_k = pred_chunks[:k]
    hits = 1.0 if any(c in gold_chunks for c in top_k) else 0.0
    # MRR/MAP computed over ranked list against gold set
    return hits

# Generation scoring
def score_generation(pred_answer, gold_answer):
    return 1.0 if pred_answer == gold_answer else 0.0

Common pitfalls

• NULL queries (301 instances) must be excluded from retrieval evaluation as they have no matching evidence.
• Retrieval performance heavily depends on chunk size (256 tokens) and context window limits; restricting top-K chunks significantly drops Hits@4.
• Generation accuracy drops sharply when using retrieved chunks vs. ground-truth evidence, highlighting retrieval bottlenecks rather than just LLM capability.

Evidence (verbatim from paper)

Table 5 shows the retrieval result of using different embedding models. It shows that there is still a significant gap in retrieving relevant evidence for the multi-hop queries. While Rerank can effectively improve retrieval relevance, the highest Hits@10 is only 0.7467 when the Reranker technique is used.

Citation

@misc{tang2024multihoprag,
  title={MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2401.15391}
}

• arXiv: 2401.15391