QA Benchmarks Eval

qa-benchmarks-eval

Stateful Evidence-Driven Retrieval-Augmented Generation with Iterative Reasoning — Qi Dong et al. (2026) (arXiv:2604.14170, 2026)

What this evaluates

Evaluates the capability of retrieval-augmented generation systems to answer complex, multi-hop, and long-form questions by iteratively retrieving, structuring, and accumulating evidence from documents.

Datasets

• StrategyQA — total ?; splits: test (-1)
• ASQA — total ?; splits: test (-1)
• NQ — total ?; splits: test (-1)
• 2WikiMultiHopQA — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)

Metrics

• EM, F1, ACC (primary) — range: [0, 1]
  • EM: 1 if predicted answer exactly matches gold answer, else 0. F1: token-level F1 score between predicted and gold answers. ACC: accuracy (proportion of exact matches).

Input / output format

Input: Question from a QA benchmark, optionally with retrieved documents injected into the context.

Output: Generated natural language answer string.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = [1.0 if pred.strip() == gold.strip() else 0.0 for pred, gold in zip(predictions, golds)]
    f1_scores = [token_f1(pred, gold) for pred, gold in zip(predictions, golds)]
    acc = sum(em_scores) / len(em_scores)
    return {"EM": sum(em_scores)/len(em_scores), "F1": sum(f1_scores)/len(f1_scores), "ACC": acc}

Common pitfalls

• The paper samples exactly 2000 instances per benchmark for ablation and noise experiments, which may not represent the full dataset distribution.
• F1 and ACC are reported alongside EM, but on long-form QA (ASQA), F1 is the primary reliability metric while ACC/EM are often near zero.
• Iteration count significantly impacts results (5 vs 6 iterations show diminishing returns), so comparisons must fix iteration depth.

Evidence (verbatim from paper)

Table 1. Performance comparison between the proposed framework and representative baselines on five QA benchmarks ... EM, F1, ACC ... The overall performance and comparison results are presented in Table 1, where the best-performing method for each metric is highlighted in bold. Our proposed framework consistently achieves the strongest performance across all benchmarks and evaluation metrics.

Citation

@misc{dong2026stateful,
  title={Stateful Evidence-Driven Retrieval-Augmented Generation with Iterative Reasoning},
  author={Qi Dong et al. (2026)},
  year={2026},
  note={arXiv:2604.14170}
}

• arXiv: 2604.14170