RAG Evaluation Matrix
Do not choose agentic RAG by fashion. Compare retrieval designs against domain questions, answerability, correctness, cost, latency, and failure reasons.
Use When
- The user asks whether to use basic RAG, enhanced RAG, GraphRAG, or agentic RAG.
- A RAG system works on demos but fails on domain-specific questions.
- You need to compare embedding models, rerankers, chunking, query rewriting, or tool orchestration.
- LLM-as-judge scores need alignment with human review.
Evaluation Flow
- Build a domain question set with answerable, unanswerable, ambiguous, multi-hop, and adversarial cases.
- For each candidate pipeline, log retrieved evidence, final answer, citations, latency, token usage, and cost.
- Score answer correctness and answerability separately.
- Add evidence quality checks: citation support, contradiction handling, source freshness, and missing-source diagnosis.
- Segment results by question type instead of reporting only one aggregate score.
- Inspect low-correctness failures and map them to retrieval, synthesis, tool orchestration, or corpus gaps.
- Pick the simplest design that meets quality, latency, and cost constraints.
Matrix
| Dimension |
Basic RAG |
Enhanced RAG |
Agentic RAG |
| Best for |
Stable FAQ and narrow corpora |
Noisy corpora, query mismatch, reranking |
Multi-step, ambiguous, tool-rich tasks |
| Main risk |
Weak recall and unsupported answers |
Pipeline complexity |
Cost, latency, loops, tool misuse |
| Eval focus |
Retrieval recall and citation support |
Component ablations |
Trajectory, action choice, stopping behavior |
| Ship gate |
Correctness and answerability meet threshold |
Ablation proves each module helps |
Agentic gains justify extra cost |
Script
Use the helper to combine per-run JSON metrics into a decision table:
python skills/agent-eval/rag-evaluation-matrix/scripts/rag_eval_matrix.py results/*.json
Expected JSON fields: pipeline, question_type, correct, answerable_correct, latency_ms, cost_usd.
Common Mistakes
| Mistake |
Fix |
| Optimizing average score only |
Break down by question type and domain |
| Judging unanswerable questions as wrong by default |
Score answerability separately |
| Skipping human calibration |
Sample judge disagreements and tune rubrics |
| Choosing agentic RAG without ablation |
Compare against enhanced RAG at equal budget |
| Ignoring failure reasons |
Classify each miss before tuning |
References
1---2name: rag-evaluation-matrix3description: Use when comparing basic, enhanced, GraphRAG, or agentic RAG designs, evaluating domain-specific RAG quality, tuning retrieval components, or deciding whether agentic RAG is worth its cost.4---56# RAG Evaluation Matrix78Do not choose agentic RAG by fashion. Compare retrieval designs against domain questions, answerability, correctness, cost, latency, and failure reasons.910## Use When1112- The user asks whether to use basic RAG, enhanced RAG, GraphRAG, or agentic RAG.13- A RAG system works on demos but fails on domain-specific questions.14- You need to compare embedding models, rerankers, chunking, query rewriting, or tool orchestration.15- LLM-as-judge scores need alignment with human review.1617## Evaluation Flow18191. Build a domain question set with answerable, unanswerable, ambiguous, multi-hop, and adversarial cases.202. For each candidate pipeline, log retrieved evidence, final answer, citations, latency, token usage, and cost.213. Score answer correctness and answerability separately.224. Add evidence quality checks: citation support, contradiction handling, source freshness, and missing-source diagnosis.235. Segment results by question type instead of reporting only one aggregate score.246. Inspect low-correctness failures and map them to retrieval, synthesis, tool orchestration, or corpus gaps.257. Pick the simplest design that meets quality, latency, and cost constraints.2627## Matrix2829| Dimension | Basic RAG | Enhanced RAG | Agentic RAG |30| --- | --- | --- | --- |31| Best for | Stable FAQ and narrow corpora | Noisy corpora, query mismatch, reranking | Multi-step, ambiguous, tool-rich tasks |32| Main risk | Weak recall and unsupported answers | Pipeline complexity | Cost, latency, loops, tool misuse |33| Eval focus | Retrieval recall and citation support | Component ablations | Trajectory, action choice, stopping behavior |34| Ship gate | Correctness and answerability meet threshold | Ablation proves each module helps | Agentic gains justify extra cost |3536## Script3738Use the helper to combine per-run JSON metrics into a decision table:3940```bash41python skills/agent-eval/rag-evaluation-matrix/scripts/rag_eval_matrix.py results/*.json42```4344Expected JSON fields: `pipeline`, `question_type`, `correct`, `answerable_correct`, `latency_ms`, `cost_usd`.4546## Common Mistakes4748| Mistake | Fix |49| --- | --- |50| Optimizing average score only | Break down by question type and domain |51| Judging unanswerable questions as wrong by default | Score answerability separately |52| Skipping human calibration | Sample judge disagreements and tune rubrics |53| Choosing agentic RAG without ablation | Compare against enhanced RAG at equal budget |54| Ignoring failure reasons | Classify each miss before tuning |5556## References5758- arXiv: Is Agentic RAG worth it? - https://arxiv.org/abs/2601.0771159- arXiv: RAGalyst - https://arxiv.org/abs/2511.0450260- Hugging Face Papers: RAGalyst - https://huggingface.co/papers/2511.0450261- Hugging Face dataset: RAGalyst QAC - https://huggingface.co/datasets/hoskerelab/ragalyst-qac
mouadja02