Crumqs Eval

crumqs-eval

Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries — Liu et al. (2025) (arXiv:2510.11956, 2025)

What this evaluates

Evaluates RAG systems on synthetic unanswerable and multi-hop queries to measure their refusal behavior, hallucination rates, and susceptibility to reasoning shortcuts when contexts are disjointed or out-of-distribution.

Datasets

• CRUMQs — total 3048; splits: test (3048)
• UAEval4RAG — total 7559; splits: test (7559)
• MultiHop-RAG — total 2556; splits: test (2556)

Metrics

• acceptable ratio — range: [0, 1]
  • Proportion of model responses deemed acceptable (i.e., correctly refusing or stating unanswerability) for unanswerable queries.
• unanswered ratio — range: [0, 1]
  • Proportion of model responses that fail to provide an answer or clarification for unanswerable queries.
• ask-for-clarification ratio — range: [0, 1]
  • Proportion of model responses that request additional information for unanswerable queries.
• accuracy — range: [0, 1]
  • Semantic equivalence between target and predicted answers, judged via LLM (Gemini-2.0-Flash).
• F1 score — range: [0, 1]
  • Average F1 for answer prediction and paragraph-level support identification across models and datasets.
• cheatability ratio (primary) — range: [0, 1]
  • Ratio of F1 scores in the DiRe probe (disjointed contexts) vs. non-probe (oracle) settings, representing the percentage of performance attributable to disconnected reasoning.

Input / output format

Input: Query alone (for unanswerability evaluation) or query paired with retrieved/disjointed document chunks (for cheatability evaluation).

Output: Generated answer text, optionally accompanied by paragraph-level support identification.

Scoring recipe

# Unanswerability metrics
acceptable = sum(1 for pred in preds if is_acceptable_refusal(pred)) / len(preds)
unanswered = sum(1 for pred in preds if is_unanswered(pred)) / len(preds)
clarification = sum(1 for pred in preds if asks_for_clarification(pred)) / len(preds)
accuracy = llm_judge_semantic_equivalence(gold, preds)

# Cheatability metric
f1_non_probe = compute_f1(gold_answers, gold_supports, oracle_preds)
f1_probe = compute_f1(gold_answers, gold_supports, disjointed_preds)
cheatability_ratio = f1_probe / f1_non_probe

Common pitfalls

• Unanswerability evaluation explicitly assumes no access to external documents, so models must rely on internal knowledge or explicit refusal.
• Cheatability relies on the DiRe probe transformation; a high ratio indicates shortcut reliance rather than robust multi-hop reasoning.
• Accuracy for unanswerable queries uses LLM-as-a-judge for semantic equivalence, which may introduce bias compared to exact-match metrics.

Evidence (verbatim from paper)

For unanswerability evaluation, we adopt the metrics of acceptable ratio, unanswered ratio, and ask-for-clarification ratio from [[29]]. We additionally score accuracy by running LLM judgments of semantic equivalence between target and predicted answers (Gemini-2.0-Flash prompted as in [[25]]). For cheatability evaluation, we compute the average F1 score for each modeldatasettask setting as in [[39], [38]]. The cheatability of each dataset is then measured as the ratio of F1 scores in the probe vs. non-probe settings, which represents the percentage of model performance attributable to disconnected reasoning.

Citation

@misc{liu2025crumqs,
  title={Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.11956}
}

• arXiv: 2510.11956