RAG Failure Diagnosis
When a RAG answer is wrong, walk the layers backward from output to root cause. Most "the LLM hallucinated" reports are actually upstream failures that surface at generation.
Critical principle: don't fix downstream when upstream is broken. Tuning the reranker won't help if the parser dropped the table.
The diagnostic chain
For any wrong answer, walk in this order. The first layer where you find a problem is your root cause — stop walking there.
Wrong answer
│
├─ Layer 9: LLM generation
│ Did the LLM ignore evidence it had?
│ check: was the correct chunk in the context window?
│
├─ Layer 8: Context packing
│ Did the right chunk reach the LLM?
│ check: was it retrieved but dropped before packing?
│
├─ Layer 7: Reranker
│ Did rerank bury the right chunk?
│ check: was the right chunk in pre-rerank candidates but not top-k?
│
├─ Layer 6: Fusion (RRF)
│ Did fusion lose a strong single-retriever hit?
│ check: was it top-3 in BM25 or dense alone but buried after RRF?
│
├─ Layer 5: Retrieval (dense)
│ Did dense find the right chunk?
│ check: similarity score; close to top hits?
│
├─ Layer 4: Retrieval (sparse / BM25)
│ Did BM25 find the right chunk?
│ check: does the chunk contain query terms? Stop-words eating query?
│
├─ Layer 3: Filtering
│ Did filters wrongly exclude the right chunk?
│ check: replay query without filters; does it appear?
│
├─ Layer 2: Chunking
│ Was the answer split across chunk boundaries?
│ check: does the answer text exist in any single chunk?
│
└─ Layer 1: Parsing / Ingestion
Was the source content captured at all?
check: does the raw document contain the answer?
Failure modes by layer
| Layer |
Symptom |
Likely cause |
Fix direction |
| 1 Parsing |
Answer not in any chunk; raw doc has it |
Parser dropped table, lost footer, OCR failed |
Better parser (Unstructured.io / LlamaParse), OCR quality gates, chunk quality scoring |
| 1 Ingestion |
Document not indexed |
Failed job, deletion bug, tombstone |
Audit ingestion logs, idempotency check |
| 2 Chunking |
Condition separated from rule |
Fixed-size split mid-statement |
Document-aware chunking, parent-child, overlap |
| 2 Chunking |
Table broken across chunks |
Naive splitter cut the table |
Adaptive chunking, preserve tables whole |
| 3 Filtering |
Right chunk filtered out |
Metadata mismatch, tenant filter too aggressive, version filter |
Audit filter logic; check effective_to / version |
| 3 Filtering |
Post-filter killed recall |
Top-k all filtered after retrieval |
Switch to pre-filtered ANN |
| 4 Sparse |
Exact term not matched |
Tokenizer issue, stop-words too aggressive |
Check tokenization; rebuild with correct analyzer |
| 5 Dense |
Paraphrase not matched |
Domain-weak embedder, chunk size wrong |
Test domain embedder; adjust chunk size |
| 5 Dense |
Proper noun missed |
Model never saw the proper noun |
Hybrid (BM25 catches it) |
| 6 Fusion |
Strong hit buried |
One retriever had 1 great hit, other had 50 mediocre |
Inspect rank-by-rank; consider weighted RRF |
| 7 Reranker |
Wrong chunk preferred |
Reranker mismatched to domain |
Try different reranker; eval on labeled pairs |
| 7 Reranker |
Length bias |
Long irrelevant beat short relevant |
Length normalization; switch reranker |
| 8 Packing |
Right chunk dropped |
Aggressive top-k, dedup removed unique answer |
Loosen top-k; smarter dedup |
| 8 Packing |
Lost in the middle |
Right chunk at end of long context |
Move highest-scored first |
| 9 Generation |
LLM ignored evidence |
Weak grounding prompt; model priors |
Strict grounding; structured output; abstention threshold |
| 9 Generation |
Hallucinated citation |
Citation present, chunk doesn't support claim |
Per-claim citation checking; answer contract validation |
Common failure patterns (signature → layer)
| Pattern |
Signature |
Likely layer |
| Plausible but wrong |
Coherent answer, no citation matches |
9 or 8 |
| "I don't know" when answer exists |
Right chunk never retrieved |
4–6 |
| Outdated answer |
Newer version exists, old retrieved |
3 (version filter) or 5 (no freshness signal) |
| Wrong tenant data |
Info from another customer |
3 — security incident, page someone |
| Repeated / redundant |
Same content cited multiple times |
8 (no dedup) |
| Right facts, wrong combination |
Each fact cited, combination invented |
9 or 2 (lost connector across chunks) |
How to apply
When the user reports a RAG failure:
Get the trace. Ask for: query, classification (if any), filters applied, all retriever outputs with scores, RRF result, rerank result, context sent to LLM, LLM response. If they don't have this — that's the first problem to fix (audit logs per §39.2).
Check Layer 1 first. Does the answer exist in any chunk? If no, stop. Nothing downstream matters. Recommend parser / ingestion fixes.
Walk forward layer by layer. At each layer, ask: was the right chunk present? The first layer where it disappears is the root cause.
Match against common patterns — they short-circuit the walk.
For tenant data leakage: page immediately. Pre-filter chokepoint is broken; this is a security incident, not a quality issue.
Anti-patterns to watch for
- "It's just hallucinating, raise the temperature lower" → almost never the answer; check upstream first
- "Add more chunks to context" → if retrieval is wrong, more wrong chunks won't help
- "Try a bigger model" → bigger model with same broken retrieval is still broken
- "Tune the reranker" before checking that the right chunk is in the candidate pool
Reference: full playbook §40 (failure taxonomy), §39.2 (audit log schema).
1---2name: rag-debugger3description: Use when a RAG system is returning bad answers — hallucinations, wrong content, missing exact terms, low recall, slow responses, stale data, citation errors, tenant data leakage, or any production retrieval failure. Walks the 9-layer diagnostic chain backward from output to root cause. Triggers on phrases like "RAG returning wrong answers", "hallucinating", "can't find document", "wrong tenant data", "low recall", "RAG is slow", "missing citations".4---56# RAG Failure Diagnosis78When a RAG answer is wrong, **walk the layers backward** from output to root cause. Most "the LLM hallucinated" reports are actually upstream failures that surface at generation.910**Critical principle:** don't fix downstream when upstream is broken. Tuning the reranker won't help if the parser dropped the table.1112## The diagnostic chain1314For any wrong answer, walk in this order. **The first layer where you find a problem is your root cause** — stop walking there.1516```17Wrong answer18 │19 ├─ Layer 9: LLM generation20 │ Did the LLM ignore evidence it had?21 │ check: was the correct chunk in the context window?22 │23 ├─ Layer 8: Context packing24 │ Did the right chunk reach the LLM?25 │ check: was it retrieved but dropped before packing?26 │27 ├─ Layer 7: Reranker28 │ Did rerank bury the right chunk?29 │ check: was the right chunk in pre-rerank candidates but not top-k?30 │31 ├─ Layer 6: Fusion (RRF)32 │ Did fusion lose a strong single-retriever hit?33 │ check: was it top-3 in BM25 or dense alone but buried after RRF?34 │35 ├─ Layer 5: Retrieval (dense)36 │ Did dense find the right chunk?37 │ check: similarity score; close to top hits?38 │39 ├─ Layer 4: Retrieval (sparse / BM25)40 │ Did BM25 find the right chunk?41 │ check: does the chunk contain query terms? Stop-words eating query?42 │43 ├─ Layer 3: Filtering44 │ Did filters wrongly exclude the right chunk?45 │ check: replay query without filters; does it appear?46 │47 ├─ Layer 2: Chunking48 │ Was the answer split across chunk boundaries?49 │ check: does the answer text exist in any single chunk?50 │51 └─ Layer 1: Parsing / Ingestion52 Was the source content captured at all?53 check: does the raw document contain the answer?54```5556## Failure modes by layer5758| Layer | Symptom | Likely cause | Fix direction |59|---|---|---|---|60| 1 Parsing | Answer not in any chunk; raw doc has it | Parser dropped table, lost footer, OCR failed | Better parser (Unstructured.io / LlamaParse), OCR quality gates, chunk quality scoring |61| 1 Ingestion | Document not indexed | Failed job, deletion bug, tombstone | Audit ingestion logs, idempotency check |62| 2 Chunking | Condition separated from rule | Fixed-size split mid-statement | Document-aware chunking, parent-child, overlap |63| 2 Chunking | Table broken across chunks | Naive splitter cut the table | Adaptive chunking, preserve tables whole |64| 3 Filtering | Right chunk filtered out | Metadata mismatch, tenant filter too aggressive, version filter | Audit filter logic; check effective_to / version |65| 3 Filtering | Post-filter killed recall | Top-k all filtered after retrieval | Switch to pre-filtered ANN |66| 4 Sparse | Exact term not matched | Tokenizer issue, stop-words too aggressive | Check tokenization; rebuild with correct analyzer |67| 5 Dense | Paraphrase not matched | Domain-weak embedder, chunk size wrong | Test domain embedder; adjust chunk size |68| 5 Dense | Proper noun missed | Model never saw the proper noun | Hybrid (BM25 catches it) |69| 6 Fusion | Strong hit buried | One retriever had 1 great hit, other had 50 mediocre | Inspect rank-by-rank; consider weighted RRF |70| 7 Reranker | Wrong chunk preferred | Reranker mismatched to domain | Try different reranker; eval on labeled pairs |71| 7 Reranker | Length bias | Long irrelevant beat short relevant | Length normalization; switch reranker |72| 8 Packing | Right chunk dropped | Aggressive top-k, dedup removed unique answer | Loosen top-k; smarter dedup |73| 8 Packing | Lost in the middle | Right chunk at end of long context | Move highest-scored first |74| 9 Generation | LLM ignored evidence | Weak grounding prompt; model priors | Strict grounding; structured output; abstention threshold |75| 9 Generation | Hallucinated citation | Citation present, chunk doesn't support claim | Per-claim citation checking; answer contract validation |7677## Common failure patterns (signature → layer)7879| Pattern | Signature | Likely layer |80|---|---|---|81| Plausible but wrong | Coherent answer, no citation matches | 9 or 8 |82| "I don't know" when answer exists | Right chunk never retrieved | 4–6 |83| Outdated answer | Newer version exists, old retrieved | 3 (version filter) or 5 (no freshness signal) |84| Wrong tenant data | Info from another customer | 3 — **security incident, page someone** |85| Repeated / redundant | Same content cited multiple times | 8 (no dedup) |86| Right facts, wrong combination | Each fact cited, combination invented | 9 or 2 (lost connector across chunks) |8788## How to apply8990When the user reports a RAG failure:91921. **Get the trace.** Ask for: query, classification (if any), filters applied, all retriever outputs with scores, RRF result, rerank result, context sent to LLM, LLM response. If they don't have this — that's the first problem to fix (audit logs per §39.2).93942. **Check Layer 1 first.** Does the answer exist in any chunk? If no, stop. Nothing downstream matters. Recommend parser / ingestion fixes.95963. **Walk forward layer by layer.** At each layer, ask: was the right chunk present? The first layer where it disappears is the root cause.97984. **Match against common patterns** — they short-circuit the walk.991005. **For tenant data leakage: page immediately.** Pre-filter chokepoint is broken; this is a security incident, not a quality issue.101102## Anti-patterns to watch for103104- "It's just hallucinating, raise the temperature lower" → almost never the answer; check upstream first105- "Add more chunks to context" → if retrieval is wrong, more wrong chunks won't help106- "Try a bigger model" → bigger model with same broken retrieval is still broken107- "Tune the reranker" before checking that the right chunk is in the candidate pool108109Reference: full playbook §40 (failure taxonomy), §39.2 (audit log schema).
kumaran-is