Cue R Eval

cue-r-eval

CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation — Jain et al. (2026) (arXiv:2604.05467, 2026)

What this evaluates

This evaluation probes the per-evidence-item utility and trace sensitivity in single-shot retrieval-augmented generation. It measures how removing, replacing, or duplicating retrieved context chunks affects answer correctness, grounding faithfulness, confidence calibration, and reasoning trace stability.

Datasets

• HotpotQA (distractor setting) — total ?; splits: test (-1)
• 2WikiMultihopQA — total 100; splits: test (100)

Metrics

• Soft Correctness (primary) — range: [0, 1]
  • Normalized match using yes/no canonicalization, number normalization, and high-overlap fuzzy matching with F1 ≥ 0.8.
• Answer F1 — range: [0, 1]
  • Token-level F1 score between the predicted answer and the gold answer.
• Grounding Score — range: [0, 1]
  • G = |{u ∈ U : title(u) ∈ S}| / |U|, where U is the set of model-used chunk IDs and S is the set of gold support titles (0 if U=∅).
• Confidence Error — range: [0, 1]
  • CE = |c - 1[is_correct]|, where c is the model's self-reported confidence and 1[is_correct] is 1 if correct, 0 otherwise.
• Trace Divergence — range: other
  • Measures the shift in model-used chunk identifiers between original and perturbed retrieval conditions. Computed via Equation 9 in the paper.

Input / output format

Input: Question text plus top-5 BM25-retrieved passage chunks (formatted with stable IDs like ctx_0, ctx_1...). In the zero-retrieval control, only the question is provided.

Output: Predicted answer string, self-reported confidence score (c), and a trace indicating which chunk identifiers (e.g., ctx_0) the model explicitly used in its reasoning.

Scoring recipe

def score(predictions, golds, traces, confidences):
    results = {'soft_correctness': [], 'answer_f1': [], 'grounding_score': [], 'confidence_error': []}
    for pred, gold, trace, conf in zip(predictions, golds, traces, confidences):
        is_correct = 1 if fuzzy_match(pred, gold, threshold=0.8) else 0
        results['soft_correctness'].append(is_correct)
        results['answer_f1'].append(token_f1(pred, gold))
        U = set(trace.used_chunk_ids)
        S = set(gold.support_titles)
        results['grounding_score'].append(len([u for u in U if u.title in S]) / len(U) if U else 0.0)
        results['confidence_error'].append(abs(conf - is_correct))
    return {k: mean(v) for k, v in results.items()}

Common pitfalls

• Grounding Score is a coarse proxy; title-level matching does not verify that the model actually used the correct information within a chunk, and gold support titles may not cover all useful evidence.
• Confidence Error measures instance-level calibration mismatch rather than distributional calibration, so it should not be interpreted as a standard calibration metric.
• Final answer correctness can remain unchanged while trace divergence and confidence error shift significantly, meaning answer-only metrics miss critical retrieval degradation.

Evidence (verbatim from paper)

We evaluate each run using five primary metrics: Soft Correctness. Normalized match with yes/no canonicalization, number normalization, and high-overlap fuzzy matching (F1 ≥ 0.8). Answer F1. Token-level F1 between the predicted answer and gold answer. Grounding Score (proxy). Fraction of model-used chunk identifiers whose titles match gold supporting facts: G=|{u∈U:title(u)∈S}|/|U|, where U is the set of used chunk IDs and S is the set of gold support titles (0 if U=∅). This is a coarse proxy: title-level matching does not verify that the model used the correct information within a chunk, and gold support titles may not exhaustively enumerate all useful evidence. Confidence Error. CE=|c−1[is_correct]|, where c is the model’s self-reported confidence. This is an instance-level proxy for calibration mismatch, not a distributional calibration metric in the classical sense. Trace Divergence. Computed via [Equation 9].

Citation

@misc{jain2026cue_r,
  title={CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation},
  author={Jain et al. (2026)},
  year={2026},
  note={arXiv:2604.05467}
}

• arXiv: 2604.05467