Amey-Thakur RAG evaluation
A RAG system fails in two distinct places: it retrieved the wrong material, or it retrieved the right material and answered badly. Evaluating end to end tells you it is broken; evaluating the stages separately tells you where.
Method
- Build a judged set from real questions. Actual user questions with the passages that answer them and an acceptable answer, assembled once and maintained.
- Measure retrieval independently. Was the answering passage in the top results? This single number isolates half the system and is the cheapest to improve (see relevance-tuning).
- Measure faithfulness separately. Given the retrieved context, is the answer supported by it? An unfaithful answer with correct retrieval is a generation problem.
- Check for the confident empty case. When nothing relevant was retrieved, does the system say so or invent an answer? This is the most damaging failure and the least measured.
- Test the questions your corpus cannot answer. A system that refuses appropriately is working correctly, and a judged set of only answerable questions never tests it.
- Re-run on every change. Chunking, embedding model, prompt, and reranker all interact, and a change that improves one stage can degrade the whole (see agent-eval-design).
- Track disagreement with human judgement. Automated scoring drifts from what users consider a good answer, so periodic human review calibrates it.
Boundaries
Evaluation measures the cases in the judged set, which ages as the corpus and the questions change. Automated faithfulness scoring is imperfect and can be gamed by hedging. Good scores on a small set do not generalise to a long tail of unusual questions.