AllanWessels Failure Clustering
When to use
Before any blue team change, read this skill to interpret
/reports/latest.json correctly. Most failing cases are not independent
problems; they cluster around a small number of root causes.
The diagnostic matrix
The judge reports both retrieval_recall@5 and faithfulness per case.
Plot the failing cases in your head:
| Low retrieval | High retrieval | |
|---|---|---|
| High faithfulness | Honest "I don't know" — index gap or ingestion gap | Working ✓ (rare in failing cases) |
| Low faithfulness | Both broken — fix retrieval first | Model ignoring context — prompt issue |
Pick which quadrant has the most cases. That tells you which layer to fix.
The four failure modes within those quadrants
Answer missing from any chunk. Diagnose by manually searching
/corpus/for the answer. If you can't find it either, the knowledge base has a real gap; nothing pipeline-side will help. If you can find it but the index doesn't have it, that's an ingestion problem (parser, OCR, filter, freshness).Right chunk exists but wasn't retrieved. Diagnose by running the failing question with a high
k(50+) and checking whether the known-good chunk appears anywhere in the results. If absent: the embedding model is failing on this query type (try HyDE, multi-query, hybrid). If present but buried: add or upgrade the reranker, or increase initial k. If only one lane found it: check hybrid merge weights.Chunk retrieved but model ignored it. Symptoms: the model cites a wrong chunk, or generates from prior knowledge despite having context. Fixes: tighten the prompt (
answer using ONLY the context), reorder chunks so the most relevant is last, require chunk-numbered citations.Invented citation. The model cites chunk [N] but chunk [N] doesn't actually contain the claim. Fix: add a post-generation citation verification pass (separate LLM call) that checks each citation. Expensive but unavoidable for high-stakes outputs.
How to write a hypothesis
A good hypothesis for the blue team looks like:
"8 failing cases cluster in
multi_hopcategory. All 8 have low retrieval recall (avg 0.2). Manual inspection shows the question needs information from two corpus passages, but the retriever returns one and ignores the second. Hypothesis: a query decomposition step that splits multi-hop questions into independent sub-queries would lift recall on this cluster without affecting single-hop cases."
A bad hypothesis looks like:
"Lots of cases are failing, let's try a bigger model."
The first is testable, scoped, and predicts a specific outcome. The second is a guess that, even if it works, teaches you nothing.
How to pick the cluster to attack
When multiple clusters exist, prefer (in order):
- The largest cluster — fixing it moves the score most
- The cluster you have the strongest hypothesis about — even if smaller, the iteration loop completes faster
- Clusters at the retrieval layer before clusters at the generation layer — retrieval is upstream and the cheaper layer to fix
If two clusters tie, pick the retrieval one.