Rgb Eval

rgb-eval

Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis — Amamou et al. (2026) (arXiv:2603.05698, 2026)

What this evaluates

Evaluates the robustness of Retrieval-Augmented Generation (RAG) systems against four key challenges: handling noisy or irrelevant retrieved documents, integrating information from multiple sources, rejecting queries when knowledge is insufficient, and detecting or correcting factual errors in counterfactual documents.

Datasets

• RGB Benchmark — total ?; splits: test (-1)

Metrics

• Accuracy (ACC) (primary) — range: [0, 1]
  • Proportion of correctly answered queries in the information integration and noise robustness tasks. Calculated as the number of correct predictions divided by the total number of instances.
• Rejection Rate — range: [0, 1]
  • Percentage of instances in the negative rejection task where the model correctly refuses to generate an answer when provided with insufficient or noisy documents.
• Error Detection Rate (ED) — range: [0, 1]
  • Percentage of counterfactual robustness instances where the model correctly identifies factual errors in the retrieved documents.
• Error Correction Rate (CR) — range: [0, 1]
  • Percentage of counterfactual robustness instances where the model not only detects errors but also generates the correct answer.

Input / output format

Input: A user query accompanied by a fixed set of 5 retrieved documents, which may contain noise, contradictions, irrelevant information, or factual errors depending on the task configuration.

Output: A generated natural language answer, or an explicit refusal/rejection when the model determines the retrieved information is insufficient or contradictory.

Scoring recipe

def score_rgb(predictions, golds, task_type):
    if task_type == 'negative_rejection':
        return sum(1 for p in predictions if p == 'refusal') / len(predictions)
    elif task_type == 'counterfactual':
        detected = [p for p in predictions if p.get('detected_error')]
        ed = len(detected) / len(predictions)
        corrected = [p for p in detected if p.get('correct_answer')]
        cr = len(corrected) / len(predictions)
        return {'ED': ed, 'CR': cr}
    else: # info_integration, noise_robustness
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
        return correct / len(predictions)

Common pitfalls

• RGB is not a general QA benchmark like HotpotQA; it specifically uses controlled noise injection and counterfactual documents to measure RAG sensitivity.
• The knowledge graph is constructed dynamically per query from only the 5 retrieved documents, not over the entire corpus, which affects latency and accuracy expectations.
• Error Detection Rate and Error Correction Rate are distinct metrics for the counterfactual task; detecting an error does not imply the model will correct it.

Evidence (verbatim from paper)

We adopt the same evaluation metrics used in the RGB benchmark [[1]]. The metrics are specifically tailored to each task in the benchmark. Accuracy (ACC) is used to assess performance in the information integration and noise robustness experiments. Rejection Rate is considered to test negative rejection capability. Thus, if provided with just noisy documents, LLMs are expected to refuse to generate a response. Error Detection Rate (ED) is used to evaluate model’s ability to identify factual errors in documents, and is applied specifically for the counterfactual robustness task. Error Correction Rate (CR) evaluates the ability of the model to give the correct answer after detecting errors in the counterfactual robustness task.

Citation

@misc{amamou2026towards,
  title={Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis},
  author={Amamou et al. (2026)},
  year={2026},
  note={arXiv:2603.05698}
}

• arXiv: 2603.05698