RAG Accuracy Skill

name: rag-accuracy version: 1.0 last_updated: 2025-12-04 description: RAG evaluation metrics - faithfulness, relevance, answer quality measurement license: MIT priority: critical triggers:

• "accuracy", "evaluation", "faithfulness", "relevance", "metrics"
• Evaluation tasks
• Quality assessment dependencies:
• langsmith-testing-SKILL.md compatibility:
• ragas: ">=0.1.0"
• langchain: ">=0.1.0" changelog:
• version: 1.0 date: 2025-12-04 changes:
  • Initial release for RAG evaluation
  • RAGAS framework integration
  • Custom metric implementations

📈 RAG Accuracy SKILL

Purpose

Measure RAG answer quality through three core metrics: Faithfulness (accuracy to context), Relevance (retrieval quality), and Answer Quality (overall usefulness).

Auto-Trigger Conditions

Activate when:

• User mentions: "accuracy", "evaluation", "faithfulness", "relevance", "metrics"
• Evaluation tasks
• Quality assessment
• A/B testing RAG variants

Core Metrics (3 Metrics)

1. Faithfulness (0-1 scale)

Definition: How accurate is the generated answer to the retrieved context?

Why Critical: Detects hallucinations (LLM making up facts not in context)

Calculation:

from ragas import faithfulness
from ragas.metrics import Faithfulness

metric = Faithfulness()

score = metric.score({
    "question": "What is RAG?",
    "answer": "RAG is Retrieval Augmented Generation, a technique...",
    "contexts": ["RAG combines retrieval with generation..."]
})

print(f"Faithfulness: {score}")  # 0.0-1.0

Interpretation:

• 0.9-1.0: Excellent (no hallucinations)
• 0.7-0.9: Good (minor inaccuracies)
• 0.5-0.7: Fair (some hallucinations)
• < 0.5: Bad (significant hallucinations)

Example:

# Good faithfulness (0.95)
question = "What is the capital of France?"
answer = "The capital of France is Paris."
contexts = ["Paris is the capital and largest city of France."]

# Bad faithfulness (0.3)
question = "What is the capital of France?"
answer = "The capital of France is London and it has 10 million people."
contexts = ["Paris is the capital and largest city of France."]
# Hallucination: London (wrong), 10 million (not in context)

2. Context Relevance (0-1 scale)

Definition: How relevant are the retrieved documents to the question?

Why Critical: Bad retrieval = bad answers (even perfect LLM can't fix)

Calculation:

from ragas.metrics import ContextRelevance

metric = ContextRelevance()

score = metric.score({
    "question": "How does vector search work?",
    "contexts": [
        "Vector search uses embeddings to find similar documents.",
        "Embeddings are numerical representations of text."
    ]
})

print(f"Context Relevance: {score}")  # 0.0-1.0

Interpretation:

• 0.9-1.0: Excellent (all docs highly relevant)
• 0.7-0.9: Good (most docs relevant)
• 0.5-0.7: Fair (some docs off-topic)
• < 0.5: Bad (retrieval failed)

Example:

# Good relevance (0.92)
question = "How do I make a cake?"
contexts = [
    "Mix flour, eggs, and sugar. Bake at 350°F for 30 minutes.",
    "Cake baking requires preheating the oven first."
]

# Bad relevance (0.2)
question = "How do I make a cake?"
contexts = [
    "The history of bread dates back to ancient Egypt.",
    "Different types of pasta include spaghetti and penne."
]

3. Answer Relevance (0-1 scale)

Definition: How well does the answer address the original question?

Why Critical: Ensures answer is on-topic (not tangential)

Calculation:

from ragas.metrics import AnswerRelevance

metric = AnswerRelevance()

score = metric.score({
    "question": "What is machine learning?",
    "answer": "Machine learning is a subset of AI that enables systems to learn from data."
})

print(f"Answer Relevance: {score}")  # 0.0-1.0

Interpretation:

• 0.9-1.0: Excellent (directly answers question)
• 0.7-0.9: Good (mostly on-topic)
• 0.5-0.7: Fair (some tangential info)
• < 0.5: Bad (doesn't answer question)

RAGAS Framework Integration

1. Installation

pip install ragas

2. Batch Evaluation

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevance,
    context_relevance,
    context_recall,
    context_precision
)

# Prepare dataset
dataset = {
    "question": [
        "What is RAG?",
        "How does retrieval work?"
    ],
    "answer": [
        "RAG is Retrieval Augmented Generation...",
        "Retrieval uses vector embeddings..."
    ],
    "contexts": [
        ["RAG combines retrieval with generation..."],
        ["Vector embeddings enable semantic search..."]
    ],
    "ground_truths": [
        ["RAG is a technique that..."],
        ["Retrieval finds relevant documents..."]
    ]
}

# Evaluate
results = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevance,
        context_relevance,
        context_recall,
        context_precision
    ]
)

print(results)

Output:

{
    'faithfulness': 0.85,
    'answer_relevance': 0.92,
    'context_relevance': 0.88,
    'context_recall': 0.90,
    'context_precision': 0.87
}

Custom Metric Implementations

1. Faithfulness (Manual)

from langchain_openai import ChatOpenAI

def calculate_faithfulness(answer: str, contexts: list[str]) -> float:
    """
    Check if answer claims are supported by context.
    """
    llm = ChatOpenAI(model="gpt-4", temperature=0)

    prompt = f"""
    Given the following context and answer, rate how faithful the answer is to the context.
    Return a score from 0.0 to 1.0.

    Context:
    {' '.join(contexts)}

    Answer:
    {answer}

    Faithfulness score (0.0-1.0):
    """

    response = llm.invoke(prompt)
    score = float(response.content.strip())

    return score

2. Relevance (Manual)

from langchain_openai import ChatOpenAI

def calculate_relevance(question: str, contexts: list[str]) -> float:
    """
    Check if retrieved contexts are relevant to question.
    """
    llm = ChatOpenAI(model="gpt-4", temperature=0)

    prompt = f"""
    Given the following question and retrieved documents, rate how relevant the documents are.
    Return a score from 0.0 to 1.0.

    Question:
    {question}

    Retrieved Documents:
    {' | '.join(contexts)}

    Relevance score (0.0-1.0):
    """

    response = llm.invoke(prompt)
    score = float(response.content.strip())

    return score

LangSmith Integration

1. Add Metrics as Feedback

from langsmith import Client
from ragas.metrics import Faithfulness, ContextRelevance

client = Client()
faithfulness_metric = Faithfulness()
relevance_metric = ContextRelevance()

# After RAG query
result = rag_chain.invoke(query)

# Calculate metrics
faithfulness_score = faithfulness_metric.score({
    "question": query,
    "answer": result["answer"],
    "contexts": [doc.page_content for doc in result["source_documents"]]
})

relevance_score = relevance_metric.score({
    "question": query,
    "contexts": [doc.page_content for doc in result["source_documents"]]
})

# Send to LangSmith
client.create_feedback(
    run_id=run_id,
    key="faithfulness",
    score=faithfulness_score
)

client.create_feedback(
    run_id=run_id,
    key="context_relevance",
    score=relevance_score
)

See: langsmith-testing-SKILL.md for trace collection

Quality Thresholds

Production Standards

Action on failure:

• Faithfulness < 0.7: Check prompt engineering, reduce temperature
• Context Relevance < 0.7: Improve retrieval (hybrid search, reranking)
• Answer Relevance < 0.7: Refine prompt, add few-shot examples

A/B Testing with Metrics

Compare Retrieval Strategies

from ragas import evaluate

# Strategy A: Semantic only
results_a = evaluate(
    dataset_a,
    metrics=[faithfulness, context_relevance]
)

# Strategy B: Hybrid (BM25 + Semantic)
results_b = evaluate(
    dataset_b,
    metrics=[faithfulness, context_relevance]
)

# Compare
print(f"Semantic only: {results_a}")
print(f"Hybrid search: {results_b}")

# Decision: Choose strategy with higher context_relevance

Compare Prompt Templates

# Template A: Simple
prompt_a = "Answer: {question}\nContext: {context}"

# Template B: Few-shot
prompt_b = """
Examples:
Q: What is X?
A: X is...

Now answer:
Q: {question}
Context: {context}
"""

# Evaluate both
results_a = evaluate(dataset_a, metrics=[faithfulness, answer_relevance])
results_b = evaluate(dataset_b, metrics=[faithfulness, answer_relevance])

# Decision: Choose template with higher faithfulness

Automated Quality Monitoring

Daily Checks

from langsmith import Client
from datetime import datetime, timedelta

client = Client()

# Get today's runs
runs = client.list_runs(
    project_name="rag-production",
    start_time=datetime.now() - timedelta(days=1)
)

# Calculate avg metrics
faithfulness_scores = []
relevance_scores = []

for run in runs:
    feedbacks = client.list_feedback(run_id=run.id)
    for fb in feedbacks:
        if fb.key == "faithfulness":
            faithfulness_scores.append(fb.score)
        elif fb.key == "context_relevance":
            relevance_scores.append(fb.score)

avg_faithfulness = sum(faithfulness_scores) / len(faithfulness_scores)
avg_relevance = sum(relevance_scores) / len(relevance_scores)

print(f"Avg Faithfulness: {avg_faithfulness:.2f}")
print(f"Avg Relevance: {avg_relevance:.2f}")

# Alert if below threshold
if avg_faithfulness < 0.7:
    print("WARNING: Faithfulness below threshold!")
if avg_relevance < 0.7:
    print("WARNING: Relevance below threshold!")

Best Practices

1. Evaluate on Representative Dataset

# Bad: Only simple questions
dataset = [
    "What is X?",
    "What is Y?"
]

# Good: Mix of simple, complex, edge cases
dataset = [
    "What is X?",                          # Simple
    "Compare X and Y",                     # Complex
    "What is the capital of Atlantis?",    # No answer in knowledge base
    ""                                     # Empty query
]

2. Track Metrics Over Time

import pandas as pd
from datetime import datetime

# Log daily metrics
metrics_log = []

def log_metrics(date, faithfulness, relevance):
    metrics_log.append({
        "date": date,
        "faithfulness": faithfulness,
        "relevance": relevance
    })

# Visualize trends
df = pd.DataFrame(metrics_log)
df.plot(x="date", y=["faithfulness", "relevance"])

3. Use Ground Truth for Validation

# Include expected answers in dataset
dataset = {
    "question": ["What is RAG?"],
    "ground_truths": [["RAG is Retrieval Augmented Generation..."]],
    # ... rest of dataset
}

# Evaluate with context_recall (how much of ground truth is retrieved)
from ragas.metrics import context_recall

results = evaluate(dataset, metrics=[context_recall])

Troubleshooting

Issue: Low faithfulness despite good context

Cause: LLM hallucinating or temperature too high

Solution:

# Reduce temperature
llm = ChatOpenAI(model="gpt-4", temperature=0)  # 0 = deterministic

# Add "stick to context" instruction
prompt = """
Use ONLY the provided context to answer. Do not use external knowledge.
If the answer is not in the context, say "I don't know based on the provided context."

Context: {context}
Question: {question}
"""

Issue: Low relevance despite good embeddings

Cause: Retrieval strategy not optimal

Solution:

# Use hybrid search (BM25 + Semantic)
from langchain.retrievers import EnsembleRetriever

retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, semantic_retriever],
    weights=[0.5, 0.5]
)

# Or add reranking
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker

compressor = CrossEncoderReranker()
retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

See: retrieval-patterns-SKILL.md

Integration with Other Skills

Workflow

RAG Query
   ↓
LangSmith Trace (langsmith-testing-SKILL.md)
   ↓
Get Result
   ↓
Calculate Metrics (THIS SKILL)
   ↓
Send Feedback to LangSmith
   ↓
Monitor Quality Thresholds

Related Files

Last Updated: 2025-12-04 Version: 1.0 Priority: CRITICAL (Core RAG quality measurement)