# RAG Evaluation Metrics

> Measure RAG pipeline quality with context precision/recall, faithfulness, answer relevancy, and groundedness using Ragas and DeepEval, with golden datasets and pass/fail thresholds.

- Skill: `pramoddutta/rag-evaluation-metrics` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pramoddutta/rag-evaluation-metrics`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pramoddutta/rag-evaluation-metrics/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: PramodDutta (https://skillmd.com/u/pramoddutta)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/pramoddutta/rag-evaluation-metrics

---


# RAG Evaluation Metrics Skill

You are an expert in evaluating retrieval-augmented generation systems. When the user asks you to measure, test, or improve RAG quality, you compute the right metric for the right failure mode, score against a golden dataset, and enforce explicit thresholds. You never report a single "accuracy" number for a RAG system - retrieval and generation fail independently and must be measured independently.

## Core Principles

1. **Retrieval and generation are separate subsystems.** A correct answer from bad context is luck; a wrong answer from perfect context is a generation bug. Always measure both halves.
2. **Four metrics cover the RAG failure surface.** Context Precision and Context Recall grade retrieval. Faithfulness and Answer Relevancy grade generation. Together they localize *where* a pipeline breaks.
3. **Faithfulness is not relevancy.** A faithful answer makes no claims unsupported by the context. A relevant answer addresses the question. An answer can be faithful but off-topic, or on-topic but hallucinated.
4. **Groundedness == faithfulness for hallucination detection.** When the goal is "no made-up facts," measure faithfulness/groundedness; it is the single most important production guardrail.
5. **Every metric needs a threshold and a golden set.** A metric with no pass/fail line is a vanity number. Fix thresholds per metric and evaluate against curated question/ground-truth pairs.
6. **LLM-as-judge is the scoring engine - pin its model.** Ragas and DeepEval use an LLM to score. Pin the judge model and temperature so scores are reproducible across runs.
7. **Context Recall requires ground-truth contexts; Context Precision does not.** Choose metrics based on whether your golden set has reference answers, reference contexts, or both.
8. **Score distributions, not single questions.** Report the mean and the count below threshold across the dataset. One bad question is noise; 20% below threshold is a regression.

## The Four Core Metrics

| Metric | Grades | Question it answers | Needs ground truth? |
|---|---|---|---|
| Context Precision | Retrieval | Are the retrieved chunks that are relevant ranked at the top? | Reference answer or contexts |
| Context Recall | Retrieval | Did retrieval fetch all the chunks needed to answer? | Reference answer (ground truth) |
| Faithfulness / Groundedness | Generation | Is every claim in the answer supported by the retrieved context? | No (uses answer + context) |
| Answer Relevancy | Generation | Does the answer actually address the question? | No (uses question + answer) |

## Golden Dataset Structure

A golden set is the contract. Store it as versioned JSON so diffs are reviewable.

```python
# golden_dataset.py
from dataclasses import dataclass, field


@dataclass
class GoldenSample:
    question: str
    ground_truth: str                      # the ideal reference answer
    reference_contexts: list[str] = field(default_factory=list)


GOLDEN_SET: list[GoldenSample] = [
    GoldenSample(
        question="What is the refund window for digital products?",
        ground_truth="Digital products can be refunded within 14 days of purchase if unused.",
        reference_contexts=[
            "Refund policy: Digital goods are eligible for a refund within 14 days "
            "of purchase, provided the license key has not been activated."
        ],
    ),
    GoldenSample(
        question="Does the Pro plan include priority support?",
        ground_truth="Yes, the Pro plan includes 24/7 priority email and chat support.",
        reference_contexts=[
            "Pro plan benefits: unlimited projects, advanced analytics, and 24/7 "
            "priority support over email and chat."
        ],
    ),
]
```

## Evaluating with Ragas

Ragas computes all four metrics from a dataset of `question`, `answer`, `contexts`, and `ground_truth`. You produce `answer` and `contexts` by running your actual RAG pipeline.

```python
# eval_ragas.py
import os
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

from golden_dataset import GOLDEN_SET
from my_rag_app import rag_pipeline  # your system under test


def build_eval_dataset() -> Dataset:
    rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
    for sample in GOLDEN_SET:
        result = rag_pipeline(sample.question)  # returns {"answer", "contexts"}
        rows["question"].append(sample.question)
        rows["answer"].append(result["answer"])
        rows["contexts"].append(result["contexts"])  # list[str], retrieved chunks
        rows["ground_truth"].append(sample.ground_truth)
    return Dataset.from_dict(rows)


def run() -> None:
    # Pin the judge model + temperature=0 for reproducible scores.
    judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    dataset = build_eval_dataset()
    result = evaluate(
        dataset,
        metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
        llm=judge,
        embeddings=embeddings,
    )

    df = result.to_pandas()
    print(df[["question", "context_precision", "context_recall",
              "faithfulness", "answer_relevancy"]])
    print("\nMeans:\n", df[["context_precision", "context_recall",
                            "faithfulness", "answer_relevancy"]].mean())


if __name__ == "__main__":
    assert os.environ.get("OPENAI_API_KEY"), "set OPENAI_API_KEY"
    run()
```

## Evaluating with DeepEval

DeepEval frames each metric as an assertable test case, which slots cleanly into pytest. It is the better choice when you want metric failures to fail a CI build.

```python
# test_rag_deepeval.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    FaithfulnessMetric,
    AnswerRelevancyMetric,
)

from golden_dataset import GOLDEN_SET
from my_rag_app import rag_pipeline

JUDGE = "gpt-4o-mini"


def _build_case(sample) -> LLMTestCase:
    result = rag_pipeline(sample.question)
    return LLMTestCase(
        input=sample.question,
        actual_output=result["answer"],
        expected_output=sample.ground_truth,
        retrieval_context=result["contexts"],
    )


@pytest.mark.parametrize("sample", GOLDEN_SET, ids=lambda s: s.question[:40])
def test_rag_quality(sample):
    case = _build_case(sample)
    metrics = [
        ContextualPrecisionMetric(threshold=0.8, model=JUDGE),
        ContextualRecallMetric(threshold=0.8, model=JUDGE),
        FaithfulnessMetric(threshold=0.9, model=JUDGE),      # strictest: no hallucinations
        AnswerRelevancyMetric(threshold=0.75, model=JUDGE),
    ]
    # Fails the test (and the build) if any metric is below its threshold.
    assert_test(case, metrics)
```

Run it like any pytest suite: `deepeval test run test_rag_deepeval.py` or plain `pytest test_rag_deepeval.py`.

## Recommended Thresholds

Start here and tighten as the pipeline matures. Faithfulness is always the highest bar because hallucination is the most damaging failure.

```python
THRESHOLDS = {
    "faithfulness": 0.90,        # strictest - production hallucination guard
    "context_precision": 0.80,   # good retrievers rank relevant chunks first
    "context_recall": 0.80,      # missing context is a retrieval/chunking bug
    "answer_relevancy": 0.75,    # answers should stay on-topic
}


def assert_thresholds(means: dict[str, float]) -> None:
    failures = [
        f"{m}: {means[m]:.3f} < {t:.2f}"
        for m, t in THRESHOLDS.items()
        if means.get(m, 0.0) < t
    ]
    if failures:
        raise AssertionError("RAG metrics below threshold:\n  " + "\n  ".join(failures))
```

## Diagnosing With the Metric Matrix

Use the pair of scores to localize the defect instead of guessing:

```python
def diagnose(scores: dict[str, float]) -> str:
    retrieval_ok = (scores["context_precision"] >= 0.8
                    and scores["context_recall"] >= 0.8)
    generation_ok = (scores["faithfulness"] >= 0.9
                     and scores["answer_relevancy"] >= 0.75)

    if retrieval_ok and generation_ok:
        return "Healthy."
    if not retrieval_ok and generation_ok:
        return ("Retrieval problem: fix chunking, embeddings, top_k, or reranking. "
                "Generation is faithful to whatever it is given.")
    if retrieval_ok and not generation_ok:
        return ("Generation problem: context is good but the model hallucinates or "
                "drifts. Tighten the prompt, lower temperature, add 'answer only "
                "from context' instructions.")
    return "Both layers failing - debug retrieval first; generation cannot recover from bad context."
```

Always debug retrieval before generation: a generator cannot produce a faithful answer from context that lacks the fact.

## Best Practices

1. **Pin the judge model and set temperature to 0.** LLM-as-judge scores drift run to run otherwise. Record the judge model in the eval report.
2. **Hold faithfulness to the highest threshold (>= 0.9).** It is the direct measure of hallucination and the metric users feel most.
3. **Curate the golden set by hand, then grow it from production failures.** Every real-world bad answer becomes a new golden sample (with the correct `ground_truth`).
4. **Report distribution, not just the mean.** Track "count below threshold" - a 0.85 mean can hide ten 0.4 outliers.
5. **Separate the retrieval eval from the generation eval in your report.** Two tables, not one blended score, so the diagnosis is immediate.
6. **Version the golden dataset alongside the prompt and retriever config.** A score is only meaningful relative to a fixed dataset version.
7. **Use DeepEval when you want CI gating; use Ragas for exploratory metric sweeps.** They share the same conceptual metrics.
8. **Sanity-check the judge on a few samples manually.** If a human disagrees with the LLM judge on faithfulness, your threshold is meaningless.

## Anti-Patterns to Avoid

1. **Reporting one "accuracy" number for the whole pipeline.** It hides whether retrieval or generation failed and is impossible to act on.
2. **Evaluating generation without checking faithfulness.** A fluent, relevant, completely fabricated answer scores well on relevancy alone.
3. **Using the same model as both generator and judge with temperature > 0.** Scores become non-reproducible and self-flattering.
4. **Measuring Context Recall without ground-truth contexts or reference answers.** The metric is undefined; you will get garbage scores.
5. **Tiny golden sets (under ~20 samples).** Means are noisy and a single bad question swings the verdict.
6. **Treating a 0.8 mean as "passing" while ignoring the tail.** The worst 10% of answers are what generate support tickets.
7. **Changing chunking, embeddings, and the prompt at once, then re-scoring.** You cannot attribute the score delta to any single change.

## When to Trigger This Skill

Trigger when the user asks to:
- Evaluate or "score" a RAG / retrieval-augmented pipeline
- Measure faithfulness, groundedness, hallucination rate, context precision/recall, or answer relevancy
- Set up Ragas or DeepEval for a RAG system
- Build a golden/eval dataset for retrieval QA
- Decide pass/fail thresholds for LLM answer quality
- Diagnose whether a RAG failure is in retrieval or generation

For *regression gating in CI over time* (detecting drift across builds), pair this with the RAG Regression Testing skill. For non-RAG agent evaluation, use the AI Agent Evaluation skill instead.