Crag Eval

crag-eval

CRAG -- Comprehensive RAG Benchmark — Xiao Yang et al. (2024) (arXiv:2406.04744, 2024)

What this evaluates

Evaluates the factual reliability and hallucination resistance of Retrieval-Augmented Generation (RAG) systems on realistic, dynamic, and long-tail questions. It measures how well models avoid generating incorrect information and appropriately abstain when knowledge is missing.

Datasets

• CRAG — total 4409; splits: validation (-1), public_test (-1), private (-1); repo https://github.com/facebookresearch/CRAG

Metrics

• truthfulness (primary) — range: [-1, 1]
  • Average score across all evaluation examples. Each response is labeled Perfect (1), Acceptable (0.5), Missing (0), or Incorrect (-1). Truthfulness = mean(scores).

Input / output format

Input: A factual question from the evaluation set.

Output: A natural language response generated by the RAG system.

Scoring recipe

def score_response(response):
    if is_perfect(response): return 1.0
    elif is_acceptable(response): return 0.5
    elif is_missing(response): return 0.0
    else: return -1.0

def compute_truthfulness(predictions, golds):
    scores = [score_response(p) for p in predictions]
    return sum(scores) / len(scores)

Common pitfalls

• Hallucinated answers are penalized (-1) while missing answers get 0, so systems should prefer abstaining over guessing.
• Automatic evaluation uses a two-step process: exact match first, then LLM judge; results are averaged across two different LLMs (ChatGPT and Llama 3) to mitigate self-preference bias.

Evidence (verbatim from paper)

We use a scoring method with score 1, 0.5, 0, and -1 for each perfect, acceptable, missing, and incorrect answer, respectively, where we penalize hallucinated answers and prefer missing answers to incorrect ones. We then define truthfulness as the average score from all examples in the evaluation set for a given RAG system.

Citation

@misc{yang2024crag,
  title={CRAG -- Comprehensive RAG Benchmark},
  author={Xiao Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.04744}
}

• arXiv: 2406.04744