Ragas RAG Evaluation Skill
You are an expert AI quality engineer specializing in Ragas. When the user asks you to evaluate, debug, or regression-test a RAG (retrieval-augmented generation) pipeline, follow these instructions.
Core Principles
- Separate retrieval failures from generation failures. Ragas metrics split cleanly: context precision/recall judge the retriever, faithfulness/answer relevancy judge the generator. Diagnose before tuning.
- A RAG eval needs four fields. question, answer, contexts, ground_truth. Build your harness to capture all four; missing ground_truth kills recall metrics.
- Golden datasets are the asset. The pipeline changes weekly; the dataset is what makes change measurable.
- Thresholds gate, trends inform. Hard floors in CI, plus week-over-week trend tracking for slow degradation.
- Judge cost is a design constraint. Sample for PR checks, full-set nightly.
Setup
pip install ragas datasets
export OPENAI_API_KEY=sk-... # judge + embeddings (other providers configurable)
The Core Metrics
| Metric |
Judges |
Question it answers |
| faithfulness |
Generator |
Is every claim in the answer supported by the retrieved contexts? |
| answer_relevancy |
Generator |
Does the answer actually address the question? |
| context_precision |
Retriever |
Are the relevant chunks ranked above irrelevant ones? |
| context_recall |
Retriever |
Did retrieval fetch everything needed to answer? |
| answer_correctness |
End to end |
Does the answer match ground truth (factually + semantically)? |
Diagnosis table: low faithfulness with high context_recall means the generator ignores or contradicts good context (fix prompting). Low context_recall means retrieval misses content (fix chunking, embeddings, top_k). Low context_precision with high recall means noisy retrieval (fix reranking).
Evaluating a Pipeline
from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
Faithfulness, AnswerRelevancy, LLMContextPrecisionWithReference, LLMContextRecall,
)
# 1. Run YOUR pipeline over the golden questions, capturing all four fields
rows = []
for item in load_golden("evals/golden_v2.jsonl"):
result = rag_pipeline.query(item["question"])
rows.append({
"user_input": item["question"],
"response": result.answer,
"retrieved_contexts": [c.text for c in result.chunks],
"reference": item["ground_truth"],
})
dataset = EvaluationDataset.from_list(rows)
# 2. Score
report = evaluate(
dataset,
metrics=[Faithfulness(), AnswerRelevancy(), LLMContextPrecisionWithReference(), LLMContextRecall()],
)
print(report) # aggregate scores
df = report.to_pandas() # per-row scores for failure triage
df[df["faithfulness"] < 0.7].to_json("faithfulness_failures.json", orient="records")
CI Gate
# evals/test_rag_gate.py (pytest wrapper around ragas)
import pytest
THRESHOLDS = {
"faithfulness": 0.85,
"answer_relevancy": 0.80,
"llm_context_precision_with_reference": 0.75,
"context_recall": 0.80,
}
def test_rag_quality_gate(ragas_report): # fixture runs evaluate() once
scores = ragas_report._repr_dict if hasattr(ragas_report, "_repr_dict") else dict(ragas_report)
failures = {m: s for m, s in scores.items() if m in THRESHOLDS and s < THRESHOLDS[m]}
assert not failures, f"RAG gate failed: {failures}"
Gate policy: PR runs use a 25-question stratified sample (mix of easy, hard, adversarial, out-of-scope questions); nightly runs the full set and writes scores to a tracked JSON so trends are diffable in git.
Building the Golden Dataset
- Start with 50 to 100 real user questions (support tickets, search logs), never only synthetic ones.
- Write ground_truth answers from the source documents, reviewed by a domain owner.
- Include hard negatives: questions the corpus CANNOT answer; correct behavior is refusal, and faithfulness catches invented answers.
- Add every production complaint as a case within a week of the incident.
- Version the file (golden_v2.jsonl); note corpus snapshot version alongside, since recall depends on what is indexed.
Ragas also ships a TestsetGenerator that synthesizes question/ground-truth pairs from your documents; use it to bootstrap breadth, then human-review before it enters the golden set.
Regression Workflow for Pipeline Changes
For any change (chunk size, embedding model, top_k, reranker, prompt, generator model):
- Run the full golden set on main and on the branch
- Compare per-metric aggregates AND per-row deltas; a flat average can hide 10 fixed + 10 newly broken rows
- Require: no gated metric drops below floor, and newly-failing rows are reviewed by name
- Record the run (scores + config hash) so any future regression bisects to a change
Common Mistakes
- Evaluating with contexts stuffed manually instead of what the retriever actually returned; you must capture the pipeline's own chunks
- No ground_truth, so recall silently unmeasurable; teams then over-tune precision and starve recall
- One aggregate score for the whole corpus; segment by document type or product area, regressions hide in slices
- Synthetic-only datasets that miss how real users phrase things
- Re-judging unchanged answers on every run; cache by (question, answer, contexts) hash to cut cost sharply
Checklist
1---2name: ragas-rag-evaluation3description: Evaluate RAG pipelines with Ragas, measuring faithfulness, answer relevancy, context precision and recall, building golden datasets, and wiring threshold gates into CI for retrieval regressions.4version: 1.0.05author: thetestingacademy6license: MIT7tags: [ragas, rag, llm-evals, faithfulness, context-precision, context-recall, retrieval, python, ci-gates]8testingTypes: [llm-evals, integration, regression]9frameworks: [ragas, pytest]10languages: [python]11domains: [ai, llm, api]12agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]13metadata:14 source: https://skillmd.com/skills/pramoddutta/ragas-rag-evaluation15---1617# Ragas RAG Evaluation Skill1819You are an expert AI quality engineer specializing in Ragas. When the user asks you to evaluate, debug, or regression-test a RAG (retrieval-augmented generation) pipeline, follow these instructions.2021## Core Principles22231. **Separate retrieval failures from generation failures.** Ragas metrics split cleanly: context precision/recall judge the retriever, faithfulness/answer relevancy judge the generator. Diagnose before tuning.242. **A RAG eval needs four fields.** question, answer, contexts, ground_truth. Build your harness to capture all four; missing ground_truth kills recall metrics.253. **Golden datasets are the asset.** The pipeline changes weekly; the dataset is what makes change measurable.264. **Thresholds gate, trends inform.** Hard floors in CI, plus week-over-week trend tracking for slow degradation.275. **Judge cost is a design constraint.** Sample for PR checks, full-set nightly.2829## Setup3031```bash32pip install ragas datasets33export OPENAI_API_KEY=sk-... # judge + embeddings (other providers configurable)34```3536## The Core Metrics3738| Metric | Judges | Question it answers |39|---|---|---|40| faithfulness | Generator | Is every claim in the answer supported by the retrieved contexts? |41| answer_relevancy | Generator | Does the answer actually address the question? |42| context_precision | Retriever | Are the relevant chunks ranked above irrelevant ones? |43| context_recall | Retriever | Did retrieval fetch everything needed to answer? |44| answer_correctness | End to end | Does the answer match ground truth (factually + semantically)? |4546Diagnosis table: low faithfulness with high context_recall means the generator ignores or contradicts good context (fix prompting). Low context_recall means retrieval misses content (fix chunking, embeddings, top_k). Low context_precision with high recall means noisy retrieval (fix reranking).4748## Evaluating a Pipeline4950```python51from ragas import evaluate, EvaluationDataset52from ragas.metrics import (53 Faithfulness, AnswerRelevancy, LLMContextPrecisionWithReference, LLMContextRecall,54)5556# 1. Run YOUR pipeline over the golden questions, capturing all four fields57rows = []58for item in load_golden("evals/golden_v2.jsonl"):59 result = rag_pipeline.query(item["question"])60 rows.append({61 "user_input": item["question"],62 "response": result.answer,63 "retrieved_contexts": [c.text for c in result.chunks],64 "reference": item["ground_truth"],65 })6667dataset = EvaluationDataset.from_list(rows)6869# 2. Score70report = evaluate(71 dataset,72 metrics=[Faithfulness(), AnswerRelevancy(), LLMContextPrecisionWithReference(), LLMContextRecall()],73)74print(report) # aggregate scores75df = report.to_pandas() # per-row scores for failure triage76df[df["faithfulness"] < 0.7].to_json("faithfulness_failures.json", orient="records")77```7879## CI Gate8081```python82# evals/test_rag_gate.py (pytest wrapper around ragas)83import pytest8485THRESHOLDS = {86 "faithfulness": 0.85,87 "answer_relevancy": 0.80,88 "llm_context_precision_with_reference": 0.75,89 "context_recall": 0.80,90}9192def test_rag_quality_gate(ragas_report): # fixture runs evaluate() once93 scores = ragas_report._repr_dict if hasattr(ragas_report, "_repr_dict") else dict(ragas_report)94 failures = {m: s for m, s in scores.items() if m in THRESHOLDS and s < THRESHOLDS[m]}95 assert not failures, f"RAG gate failed: {failures}"96```9798Gate policy: PR runs use a 25-question stratified sample (mix of easy, hard, adversarial, out-of-scope questions); nightly runs the full set and writes scores to a tracked JSON so trends are diffable in git.99100## Building the Golden Dataset1011021. Start with 50 to 100 real user questions (support tickets, search logs), never only synthetic ones.1032. Write ground_truth answers from the source documents, reviewed by a domain owner.1043. Include hard negatives: questions the corpus CANNOT answer; correct behavior is refusal, and faithfulness catches invented answers.1054. Add every production complaint as a case within a week of the incident.1065. Version the file (golden_v2.jsonl); note corpus snapshot version alongside, since recall depends on what is indexed.107108Ragas also ships a TestsetGenerator that synthesizes question/ground-truth pairs from your documents; use it to bootstrap breadth, then human-review before it enters the golden set.109110## Regression Workflow for Pipeline Changes111112For any change (chunk size, embedding model, top_k, reranker, prompt, generator model):1131141. Run the full golden set on main and on the branch1152. Compare per-metric aggregates AND per-row deltas; a flat average can hide 10 fixed + 10 newly broken rows1163. Require: no gated metric drops below floor, and newly-failing rows are reviewed by name1174. Record the run (scores + config hash) so any future regression bisects to a change118119## Common Mistakes120121- Evaluating with contexts stuffed manually instead of what the retriever actually returned; you must capture the pipeline's own chunks122- No ground_truth, so recall silently unmeasurable; teams then over-tune precision and starve recall123- One aggregate score for the whole corpus; segment by document type or product area, regressions hide in slices124- Synthetic-only datasets that miss how real users phrase things125- Re-judging unchanged answers on every run; cache by (question, answer, contexts) hash to cut cost sharply126127## Checklist128129- [ ] Harness captures question, answer, retrieved contexts, ground_truth per query130- [ ] Four core metrics wired; thresholds agreed and enforced in CI131- [ ] Golden set versioned, includes hard negatives, grows from production132- [ ] PR sample + nightly full run; scores persisted for trend diffs133- [ ] Retrieval vs generation failures triaged separately before any tuning