Repo RAG

symbols "UserAuthentication"

Semantic Search:

search "authentication middleware logic"

RAG Evaluation

Overview

Systematic evaluation of RAG quality using retrieval and end-to-end metrics. Based on Claude Cookbooks patterns.

Evaluation Metrics

Retrieval Metrics (from .claude/evaluation/retrieval_metrics.py):

• Precision: Proportion of retrieved chunks that are actually relevant
  • Formula: Precision = True Positives / Total Retrieved
  • High precision (0.8-1.0): System retrieves mostly relevant items
• Recall: Completeness of retrieval - how many relevant items were found
  • Formula: Recall = True Positives / Total Correct
  • High recall (0.8-1.0): System finds most relevant items
• F1 Score: Harmonic mean of precision and recall
  • Formula: F1 = 2 × (Precision × Recall) / (Precision + Recall)
  • Balanced measure when both precision and recall matter
• MRR (Mean Reciprocal Rank): Measures ranking quality
  • Formula: MRR = 1 / rank of first correct item
  • High MRR (0.8-1.0): Correct items ranked first

End-to-End Metrics (from .claude/evaluation/end_to_end_eval.py):

• Accuracy (LLM-as-Judge): Overall correctness using Claude evaluation
  • Compares generated answer to correct answer
  • Focuses on substance and meaning, not exact wording
  • Checks for completeness and absence of contradictions

Evaluation Process

1. Create Evaluation Dataset:

   {
     "query": "How is user authentication implemented?",
     "correct_chunks": ["src/auth/middleware.ts", "src/auth/types.ts"],
     "correct_answer": "User authentication uses JWT tokens...",
     "category": "authentication"
   }
   

2. Run Retrieval Evaluation:

   # Using Promptfoo
   npx promptfoo@latest eval -c .claude/evaluation/promptfoo_configs/rag_config.yaml
   
   # Or using Python directly
   from .claude.evaluation.retrieval_metrics import evaluate_retrieval
   metrics = evaluate_retrieval(retrieved_chunks, correct_chunks)
   print(f"Precision: {metrics['precision']}, Recall: {metrics['recall']}, F1: {metrics['f1']}, MRR: {metrics['mrr']}")
   

3. Run End-to-End Evaluation:

   # Using Promptfoo
   npx promptfoo@latest eval -c .claude/evaluation/promptfoo_configs/rag_config.yaml
   
   # Or using Python directly
   from .claude.evaluation.end_to_end_eval import evaluate_end_to_end
   result = evaluate_end_to_end(query, generated_answer, correct_answer)
   print(f"Correct: {result['is_correct']}, Explanation: {result['explanation']}")
   

Expected Performance

Based on Claude Cookbooks results:

• Basic RAG: Precision 0.43, Recall 0.66, F1 0.52, MRR 0.74, Accuracy 71%
• With Re-ranking: Precision 0.44, Recall 0.69, F1 0.54, MRR 0.87, Accuracy 81%

Best Practices

1. Separate Evaluation: Evaluate retrieval and end-to-end separately
2. Create Comprehensive Datasets: Cover common and edge cases
3. Evaluate Regularly: Run evaluations after codebase changes
4. Track Metrics Over Time: Monitor improvements
5. Use Both Metrics: Precision/Recall for retrieval, Accuracy for end-to-end

References

• RAG Patterns Guide - Implementation patterns
• Retrieval Metrics - Metric calculations
• End-to-End Evaluation - LLM-as-judge
• Evaluation Guide - Comprehensive evaluation guide