Wiki Eval Eval

wiki-eval-eval

Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al. (2023) (arXiv:2309.15217, 2023)

What this evaluates

Measures how well automated RAG scoring metrics align with human preferences in pairwise comparison tasks. It probes the ability of reference-free faithfulness, answer relevance, and context relevance estimators to replicate human judgment on answer and context quality.

Datasets

• WikiEval — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of instances where the model's preferred answer or context (based on the highest estimated metric score) matches the human annotator's preference.

Input / output format

Input: A question, two candidate answers (or two context fragments), and the corresponding retrieved context. The model computes scores for faithfulness, answer relevance, or context relevance.

Output: A binary preference label indicating which candidate is preferred, or a 0-10 score for baseline methods. The evaluation protocol outputs a binary agreement decision per instance.

Scoring recipe

def compute_agreement(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Ties in scoring are broken randomly, introducing variance in agreement rates.
• Context relevance is the hardest dimension to evaluate, often yielding lower agreement, especially for longer contexts.

Evidence (verbatim from paper)

Each WikiEval instance requires the model to compare two answers or two context fragments. We count how often the answer/context preferred by the model (i.e. with highest estimated faithfulness, answer relevance, or context relevance) coincides with the answer/context preferred by the human annotators. We report the results in terms of accuracy (i.e. the fraction of instances on which the model agrees with the annotators).

Citation

@misc{es2023ragas,
  title={Ragas: Automated Evaluation of Retrieval Augmented Generation},
  author={Es et al. (2023)},
  year={2023},
  note={arXiv:2309.15217}
}

• arXiv: 2309.15217