RAG Evaluation Metrics

RAG Evaluation Metrics Skill

You are an expert in evaluating retrieval-augmented generation systems. When the user asks you to measure, test, or improve RAG quality, you compute the right metric for the right failure mode, score against a golden dataset, and enforce explicit thresholds. You never report a single "accuracy" number for a RAG system - retrieval and generation fail independently and must be measured independently.

Core Principles

1. Retrieval and generation are separate subsystems. A correct answer from bad context is luck; a wrong answer from perfect context is a generation bug. Always measure both halves.
2. Four metrics cover the RAG failure surface. Context Precision and Context Recall grade retrieval. Faithfulness and Answer Relevancy grade generation. Together they localize where a pipeline breaks.
3. Faithfulness is not relevancy. A faithful answer makes no claims unsupported by the context. A relevant answer addresses the question. An answer can be faithful but off-topic, or on-topic but hallucinated.
4. Groundedness == faithfulness for hallucination detection. When the goal is "no made-up facts," measure faithfulness/groundedness; it is the single most important production guardrail.
5. Every metric needs a threshold and a golden set. A metric with no pass/fail line is a vanity number. Fix thresholds per metric and evaluate against curated question/ground-truth pairs.
6. LLM-as-judge is the scoring engine - pin its model. Ragas and DeepEval use an LLM to score. Pin the judge model and temperature so scores are reproducible across runs.
7. Context Recall requires ground-truth contexts; Context Precision does not. Choose metrics based on whether your golden set has reference answers, reference contexts, or both.
8. Score distributions, not single questions. Report the mean and the count below threshold across the dataset. One bad question is noise; 20% below threshold is a regression.

The Four Core Metrics

Golden Dataset Structure

A golden set is the contract. Store it as versioned JSON so diffs are reviewable.

# golden_dataset.py
from dataclasses import dataclass, field


@dataclass
class GoldenSample:
    question: str
    ground_truth: str                      # the ideal reference answer
    reference_contexts: list[str] = field(default_factory=list)


GOLDEN_SET: list[GoldenSample] = [
    GoldenSample(
        question="What is the refund window for digital products?",
        ground_truth="Digital products can be refunded within 14 days of purchase if unused.",
        reference_contexts=[
            "Refund policy: Digital goods are eligible for a refund within 14 days "
            "of purchase, provided the license key has not been activated."
        ],
    ),
    GoldenSample(
        question="Does the Pro plan include priority support?",
        ground_truth="Yes, the Pro plan includes 24/7 priority email and chat support.",
        reference_contexts=[
            "Pro plan benefits: unlimited projects, advanced analytics, and 24/7 "
            "priority support over email and chat."
        ],
    ),
]

Evaluating with Ragas

Ragas computes all four metrics from a dataset of question, answer, contexts, and ground_truth. You produce answer and contexts by running your actual RAG pipeline.

# eval_ragas.py
import os
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

from golden_dataset import GOLDEN_SET
from my_rag_app import rag_pipeline  # your system under test


def build_eval_dataset() -> Dataset:
    rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
    for sample in GOLDEN_SET:
        result = rag_pipeline(sample.question)  # returns {"answer", "contexts"}
        rows["question"].append(sample.question)
        rows["answer"].append(result["answer"])
        rows["contexts"].append(result["contexts"])  # list[str], retrieved chunks
        rows["ground_truth"].append(sample.ground_truth)
    return Dataset.from_dict(rows)


def run() -> None:
    # Pin the judge model + temperature=0 for reproducible scores.
    judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    dataset = build_eval_dataset()
    result = evaluate(
        dataset,
        metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
        llm=judge,
        embeddings=embeddings,
    )

    df = result.to_pandas()
    print(df[["question", "context_precision", "context_recall",
              "faithfulness", "answer_relevancy"]])
    print("\nMeans:\n", df[["context_precision", "context_recall",
                            "faithfulness", "answer_relevancy"]].mean())


if __name__ == "__main__":
    assert os.environ.get("OPENAI_API_KEY"), "set OPENAI_API_KEY"
    run()

Evaluating with DeepEval

DeepEval frames each metric as an assertable test case, which slots cleanly into pytest. It is the better choice when you want metric failures to fail a CI build.

# test_rag_deepeval.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    FaithfulnessMetric,
    AnswerRelevancyMetric,
)

from golden_dataset import GOLDEN_SET
from my_rag_app import rag_pipeline

JUDGE = "gpt-4o-mini"


def _build_case(sample) -> LLMTestCase:
    result = rag_pipeline(sample.question)
    return LLMTestCase(
        input=sample.question,
        actual_output=result["answer"],
        expected_output=sample.ground_truth,
        retrieval_context=result["contexts"],
    )


@pytest.mark.parametrize("sample", GOLDEN_SET, ids=lambda s: s.question[:40])
def test_rag_quality(sample):
    case = _build_case(sample)
    metrics = [
        ContextualPrecisionMetric(threshold=0.8, model=JUDGE),
        ContextualRecallMetric(threshold=0.8, model=JUDGE),
        FaithfulnessMetric(threshold=0.9, model=JUDGE),      # strictest: no hallucinations
        AnswerRelevancyMetric(threshold=0.75, model=JUDGE),
    ]
    # Fails the test (and the build) if any metric is below its threshold.
    assert_test(case, metrics)

Run it like any pytest suite: deepeval test run test_rag_deepeval.py or plain pytest test_rag_deepeval.py.

Recommended Thresholds

Start here and tighten as the pipeline matures. Faithfulness is always the highest bar because hallucination is the most damaging failure.

THRESHOLDS = {
    "faithfulness": 0.90,        # strictest - production hallucination guard
    "context_precision": 0.80,   # good retrievers rank relevant chunks first
    "context_recall": 0.80,      # missing context is a retrieval/chunking bug
    "answer_relevancy": 0.75,    # answers should stay on-topic
}


def assert_thresholds(means: dict[str, float]) -> None:
    failures = [
        f"{m}: {means[m]:.3f} < {t:.2f}"
        for m, t in THRESHOLDS.items()
        if means.get(m, 0.0) < t
    ]
    if failures:
        raise AssertionError("RAG metrics below threshold:\n  " + "\n  ".join(failures))

Diagnosing With the Metric Matrix

Use the pair of scores to localize the defect instead of guessing:

def diagnose(scores: dict[str, float]) -> str:
    retrieval_ok = (scores["context_precision"] >= 0.8
                    and scores["context_recall"] >= 0.8)
    generation_ok = (scores["faithfulness"] >= 0.9
                     and scores["answer_relevancy"] >= 0.75)

    if retrieval_ok and generation_ok:
        return "Healthy."
    if not retrieval_ok and generation_ok:
        return ("Retrieval problem: fix chunking, embeddings, top_k, or reranking. "
                "Generation is faithful to whatever it is given.")
    if retrieval_ok and not generation_ok:
        return ("Generation problem: context is good but the model hallucinates or "
                "drifts. Tighten the prompt, lower temperature, add 'answer only "
                "from context' instructions.")
    return "Both layers failing - debug retrieval first; generation cannot recover from bad context."

Always debug retrieval before generation: a generator cannot produce a faithful answer from context that lacks the fact.

Best Practices

1. Pin the judge model and set temperature to 0. LLM-as-judge scores drift run to run otherwise. Record the judge model in the eval report.
2. Hold faithfulness to the highest threshold (>= 0.9). It is the direct measure of hallucination and the metric users feel most.
3. Curate the golden set by hand, then grow it from production failures. Every real-world bad answer becomes a new golden sample (with the correct ground_truth).
4. Report distribution, not just the mean. Track "count below threshold" - a 0.85 mean can hide ten 0.4 outliers.
5. Separate the retrieval eval from the generation eval in your report. Two tables, not one blended score, so the diagnosis is immediate.
6. Version the golden dataset alongside the prompt and retriever config. A score is only meaningful relative to a fixed dataset version.
7. Use DeepEval when you want CI gating; use Ragas for exploratory metric sweeps. They share the same conceptual metrics.
8. Sanity-check the judge on a few samples manually. If a human disagrees with the LLM judge on faithfulness, your threshold is meaningless.

Anti-Patterns to Avoid

1. Reporting one "accuracy" number for the whole pipeline. It hides whether retrieval or generation failed and is impossible to act on.
2. Evaluating generation without checking faithfulness. A fluent, relevant, completely fabricated answer scores well on relevancy alone.
3. Using the same model as both generator and judge with temperature > 0. Scores become non-reproducible and self-flattering.
4. Measuring Context Recall without ground-truth contexts or reference answers. The metric is undefined; you will get garbage scores.
5. Tiny golden sets (under ~20 samples). Means are noisy and a single bad question swings the verdict.
6. Treating a 0.8 mean as "passing" while ignoring the tail. The worst 10% of answers are what generate support tickets.
7. Changing chunking, embeddings, and the prompt at once, then re-scoring. You cannot attribute the score delta to any single change.

When to Trigger This Skill

Trigger when the user asks to:

• Evaluate or "score" a RAG / retrieval-augmented pipeline
• Measure faithfulness, groundedness, hallucination rate, context precision/recall, or answer relevancy
• Set up Ragas or DeepEval for a RAG system
• Build a golden/eval dataset for retrieval QA
• Decide pass/fail thresholds for LLM answer quality
• Diagnose whether a RAG failure is in retrieval or generation

For regression gating in CI over time (detecting drift across builds), pair this with the RAG Regression Testing skill. For non-RAG agent evaluation, use the AI Agent Evaluation skill instead.