# Cue R Eval

> This evaluation probes the per-evidence-item utility and trace sensitivity in single-shot retrieval-augmented generation. It measures how removing, replacing, or duplicating retrieved context chunks affects answer correctness, grounding faithfulness, confidence calibration, and reasoning trace stability. Use when the user wants to benchmark on HotpotQA (distractor setting), 2WikiMultihopQA, or asks about evaluating this task. Reports Soft Correctness.

- Skill: `qhjqhj00/cue-r-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/cue-r-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/cue-r-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/cue-r-eval

---


# cue-r-eval

> CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation — Jain et al. (2026) (arXiv:2604.05467, 2026)

## What this evaluates

This evaluation probes the per-evidence-item utility and trace sensitivity in single-shot retrieval-augmented generation. It measures how removing, replacing, or duplicating retrieved context chunks affects answer correctness, grounding faithfulness, confidence calibration, and reasoning trace stability.

## Datasets

- **HotpotQA (distractor setting)** — total ?; splits: test (-1)
- **2WikiMultihopQA** — total 100; splits: test (100)

## Metrics

- `Soft Correctness` **(primary)** — range: [0, 1]
  - Normalized match using yes/no canonicalization, number normalization, and high-overlap fuzzy matching with F1 ≥ 0.8.
- `Answer F1` — range: [0, 1]
  - Token-level F1 score between the predicted answer and the gold answer.
- `Grounding Score` — range: [0, 1]
  - G = |{u ∈ U : title(u) ∈ S}| / |U|, where U is the set of model-used chunk IDs and S is the set of gold support titles (0 if U=∅).
- `Confidence Error` — range: [0, 1]
  - CE = |c - 1[is_correct]|, where c is the model's self-reported confidence and 1[is_correct] is 1 if correct, 0 otherwise.
- `Trace Divergence` — range: other
  - Measures the shift in model-used chunk identifiers between original and perturbed retrieval conditions. Computed via Equation 9 in the paper.

## Input / output format

**Input**: Question text plus top-5 BM25-retrieved passage chunks (formatted with stable IDs like ctx_0, ctx_1...). In the zero-retrieval control, only the question is provided.

**Output**: Predicted answer string, self-reported confidence score (c), and a trace indicating which chunk identifiers (e.g., ctx_0) the model explicitly used in its reasoning.

## Scoring recipe

```python
def score(predictions, golds, traces, confidences):
    results = {'soft_correctness': [], 'answer_f1': [], 'grounding_score': [], 'confidence_error': []}
    for pred, gold, trace, conf in zip(predictions, golds, traces, confidences):
        is_correct = 1 if fuzzy_match(pred, gold, threshold=0.8) else 0
        results['soft_correctness'].append(is_correct)
        results['answer_f1'].append(token_f1(pred, gold))
        U = set(trace.used_chunk_ids)
        S = set(gold.support_titles)
        results['grounding_score'].append(len([u for u in U if u.title in S]) / len(U) if U else 0.0)
        results['confidence_error'].append(abs(conf - is_correct))
    return {k: mean(v) for k, v in results.items()}
```

## Common pitfalls

- Grounding Score is a coarse proxy; title-level matching does not verify that the model actually used the correct information within a chunk, and gold support titles may not cover all useful evidence.
- Confidence Error measures instance-level calibration mismatch rather than distributional calibration, so it should not be interpreted as a standard calibration metric.
- Final answer correctness can remain unchanged while trace divergence and confidence error shift significantly, meaning answer-only metrics miss critical retrieval degradation.

## Evidence (verbatim from paper)

> We evaluate each run using five primary metrics: Soft Correctness. Normalized match with yes/no canonicalization, number normalization, and high-overlap fuzzy matching (F1 ≥ 0.8). Answer F1. Token-level F1 between the predicted answer and gold answer. Grounding Score (proxy). Fraction of model-used chunk identifiers whose titles match gold supporting facts: G=|{u∈U:title(u)∈S}|/|U|, where U is the set of used chunk IDs and S is the set of gold support titles (0 if U=∅). This is a coarse proxy: title-level matching does not verify that the model used the correct information within a chunk, and gold support titles may not exhaustively enumerate all useful evidence. Confidence Error. CE=|c−1[is_correct]|, where c is the model’s self-reported confidence. This is an instance-level proxy for calibration mismatch, not a distributional calibration metric in the classical sense. Trace Divergence. Computed via [Equation 9].

## Citation

```bibtex
@misc{jain2026cue_r,
  title={CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation},
  author={Jain et al. (2026)},
  year={2026},
  note={arXiv:2604.05467}
}
```

- arXiv: 2604.05467