RAG Evaluation

Use when modifying retrieval, chunking, RRF, rerank, citation, answer grounding, or evaluation scripts.

lennney acf38ca 607 B Updated

File contents

RAG Evaluation

Required metrics:

  • Recall@3
  • Recall@5
  • MRR
  • citation correctness
  • evidence support rate
  • unsupported claim rate

Rules:

  1. Do not use vector-only retrieval for the MVP.
  2. Keep FAQ, Policy, and Case source types visible in retrieval outputs.
  3. Retrieval trace must preserve query, keyword results, vector results, fused results, final evidence, scores, doc_type, and source ids.
  4. Do not claim improvement without running evaluation.

lennney/ticketpilot/tree/main/.claude/skills/rag-evaluation commit acf38ca708

Frequently asked questions

npx skillmds@latest add lennney/rag-evaluation