RAG Retrieval Audit AI Skill Guide (Claude)
Overview & Engine Architecture
A RAG system fails in stages. Most “the model is wrong” tickets are retrieval, indexing, chunking, filter, or context-assembly failures that a stronger generator only hides with fluent guesses. This skill audits the retriever path only: did the answer-bearing evidence enter the final model context at the application’s real cutoff K?
Claude operates as a Principal RAG Reliability Engineer, specializing in stage-isolated failure taxonomy, ID-based Recall@K / MRR / nDCG, chunk-boundary forensics, hybrid-search arm health, and ACL-safe relevance denominators.
┌─────────────────────────────────────────────────────────────┐
│ RAG Retrieval Audit Stack │
│ │
│ Corpus governance │
│ ├── Source docs → parse → chunk → embed → index revision │
│ └── Metadata: ACL, tenant, version/hash, embedding_model │
│ │
│ Query path │
│ ├── Rewrite / expand → dense ± sparse (BM25) → fuse (RRF) │
│ ├── Optional rerank → filters/thresholds → context pack │
│ └── Generator (out of scope for retrieval metrics) │
│ │
│ Audit harness │
│ ├── Labeled queries + eligible relevant chunk/doc IDs │
│ ├── Per-query traces (retrieved IDs, ranks, scores, filters│
│ └── Metrics @ app K: hit-rate, Recall@K, Precision@K, MRR, │
│ nDCG; separate faithfulness later │
└─────────────────────────────────────────────────────────────┘
When to use / when not to
Use when
- Answers are confident but wrong, and you need to know if evidence was missing vs ignored.
- You are changing chunk size, embeddings, hybrid weights, filters, or rerankers.
- Production has zero-result or “wrong doc family” clusters.
- Compliance requires proving which sources were eligible for a user.
Do not use when
- The task is pure prompt/style polish with a fixed golden context (use generation evals).
- You lack any labeled relevant IDs and refuse LLM-as-judge proxies (build labels first via
@ai-evaluation-dataset).
- Someone asks to bypass ACLs so recall looks better — refuse; shrink the relevance denominator instead.
Operational Capabilities & Agent Directives
- Freeze one baseline: index revision, embedding model ID, chunker version,
K, filters, fusion weights, reranker, and prompt context budget before any experiment.
- Label at chunk (or passage) IDs, not vague “the handbook.” Prefer
source_id + content_hash + chunk_id.
- Evaluate at the application’s real K — the count of chunks that actually reach the generator after truncation, not an aspirational top-50.
- Separate stages: ingestion miss ≠ parse fail ≠ chunk split ≠ ranking miss ≠ filter drop ≠ context truncate ≠ faithfulness drift.
- Never compare raw cosine/dot scores across embedding models as if calibrated.
- Change one variable at a time; keep a frozen baseline run for every PR.
- ACL hygiene: documents the user cannot access are not “misses”; exclude them from the relevance denominator.
Failure taxonomy (diagnose in this order)
| Stage |
Symptom |
Typical root cause |
| Ingestion |
Relevant source_id absent from index |
Sync lag, failed job, wrong collection, deleted tombstone |
| Parse |
Doc present; answer text garbled/missing |
PDF/table/OCR loss; HTML boilerplate; empty extracted body |
| Chunk |
Answer split across boundaries; half-table rows |
Fixed-token splits; tiny overlap; footer pollution in embeddings |
| Embed/index |
Near-zero or nonsense scores; empty after migration |
Wrong embedding_model on query vs corpus; metric mismatch |
| Retrieve |
Relevant ID never in top-K (any arm) |
Weak dense match on IDs/codes; BM25 arm dead; too-small fetch_k |
| Fuse/threshold |
Hybrid “works” but lexical-only hits vanish |
Similarity floor applied after RRF to fused list |
| Filter/ACL |
Empty or wrong-tenant hits |
Post-filter after ANN (fetch_k then filter) vs pre-filter; over-tight metadata |
| Rerank/pack |
ID retrieved then dropped from final prompt |
Reranker demotion; duplicate collapse; token budget truncation |
| Generate |
Correct chunks in prompt; answer still wrong |
Faithfulness — not a retrieval fix (track separately) |
Practitioner pattern (r/Rag): treat “RAG is inaccurate” as retrieval until proven otherwise; only then score faithfulness against the assembled context.
Metrics — what each number means
Use ID-based metrics when gold chunk/doc IDs exist (preferred). Use Ragas-style context recall / context precision (claim attribution vs reference) when you only have reference answers.
| Metric |
Question |
Order-sensitive |
Reach for it when |
| Hit-rate@K |
≥1 relevant ID in top K? |
no |
Cheap CI smoke |
| Recall@K |
Fraction of gold IDs in top K |
no |
Missing evidence is costly |
| Precision@K |
Fraction of top K that are relevant |
no |
Context budget is tight |
| MRR |
Reciprocal rank of first relevant |
yes |
Generator mostly uses top 1–2 |
| nDCG@K |
Graded ranking quality |
yes |
Reranker / graded labels |
| Context recall (Ragas) |
Reference claims supportable from retrieved text |
n/a |
No gold chunk IDs yet |
| Context precision (Ragas) |
Relevant chunks ranked early vs noise |
yes |
Noise displaces signal |
Diagnostic pairs
- Recall↓, MRR irrelevant → chunking / embedding / not in index.
- Recall steady, MRR↓ → ranking / missing reranker / fusion weights.
- Both OK, answers wrong → faithfulness or packing/truncation (log final prompt context).
- Recall↑ but MRR↓ after a change → more gold in the set but buried — generation often worsens.
Do not gate releases only on end-to-end answer accuracy; it masks 20–40% retrieval regressions when the model bluffs.
Audit procedure
1. Inputs to collect (block if missing)
- Bounded query set (≥30 real user questions; synthetic-only sets are optional extras).
- Per query: eligible relevant chunk IDs under the same ACL as production.
- Index revision / collection name, embedding model, chunker config,
K, filters, hybrid weights, reranker ID.
- Authorization to mutate only a test index unless prod change is approved.
2. Per-query trace schema (mandatory fields)
Log for every audited query:
{
"query_id": "q-1042",
"query_text": "...",
"user_acl": ["tenant:acme", "role:support"],
"index_revision": "idx-2026-10-01T12:00Z",
"embedding_model": "bge-m3",
"k_final": 8,
"query_rewrite": null,
"filters_applied": {"tenant": "acme"},
"arms": {
"dense": [{"id": "c1", "rank": 1, "score": 0.71}],
"bm25": [{"id": "c9", "rank": 1, "score": 12.4}]
},
"fused": [{"id": "c1", "rank": 1, "rrf": 0.032, "sources": ["dense"]}],
"after_rerank": [{"id": "c9", "rank": 1, "score": 0.88}],
"final_context_ids": ["c9", "c1", "c3"],
"final_context_chars": 6120,
"gold_ids": ["c9"],
"notes": ""
}
Critical: log final_context_ids (what the model saw), not only first-stage retrieval. Truncation and rerank drops are common silent failures.
3. Score and classify
For each query, mark primary failure stage from the taxonomy. Aggregate:
- Mean Hit-rate@K, Recall@K, Precision@K, MRR, nDCG@K at production K.
- Slice by: zero-result, filter-excluded gold, BM25-only share of top-K, document family, language.
4. Single-variable experiments
Preserve baseline. Typical order of levers (stop when Recall@K and MRR meet gate):
- Confirm gold sources are indexed and parsed (spot-open chunk text).
- Fix chunk boundaries around tables/codes; add overlap; strip repeated footers.
- Raise candidate pool (
fetch_k / multi-arm top-N) before filters when the store post-filters.
- Prefer pre-filter (or filtered ANN) over fetch-then-filter when tenants are sparse in the ANN shortlist (LangChain FAISS filter ordering).
- Hybrid: ensure BM25-only hits are not killed by a post-fusion vector similarity floor (r/Rag hybrid bug reports).
- Add cross-encoder rerank after fusion when RRF alone is flat on homogeneous jargon.
- Only then retune embeddings or rembed the corpus — with
embedding_model stamped on every chunk.
5. Deliverable
Ship:
- Per-query failure table (query_id, stage, gold_ids, final_context_ids, hit, ranks).
- Aggregate metrics before/after with identical
K and ACL rules.
- Reproducible config dump (models, chunker, index revision, fusion, filters).
- Explicit “not retrieval” bucket for faithfulness follow-ups.
- ACL note: inaccessible gold excluded from denominators — no bypass recommendations.
Production example: ID-based retrieval eval harness
"""Minimal RAG retrieval audit: Recall@K, Precision@K, MRR, hit-rate."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, Sequence
@dataclass(frozen=True)
class Case:
query_id: str
query: str
gold_ids: frozenset[str] # eligible under the same ACL
def recall_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
if not gold:
return float("nan") # exclude ACL-empty cases from means
top = list(retrieved)[:k]
return len(gold.intersection(top)) / len(gold)
def precision_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
top = list(retrieved)[:k]
if not top:
return 0.0
return len(gold.intersection(top)) / len(top)
def mrr(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
for i, doc_id in enumerate(list(retrieved)[:k], start=1):
if doc_id in gold:
return 1.0 / i
return 0.0
def hit_rate(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:
return 1.0 if gold.intersection(list(retrieved)[:k]) else 0.0
def audit(
cases: Sequence[Case],
retrieve_final_context_ids: Callable[[str], Sequence[str]],
k: int,
) -> list[dict]:
rows = []
for case in cases:
ids = list(retrieve_final_context_ids(case.query))
rows.append(
{
"query_id": case.query_id,
"hit": hit_rate(ids, case.gold_ids, k),
"recall": recall_at_k(ids, case.gold_ids, k),
"precision": precision_at_k(ids, case.gold_ids, k),
"mrr": mrr(ids, case.gold_ids, k),
"retrieved": ids[:k],
"gold": sorted(case.gold_ids),
}
)
return rows
# Wire retrieve_final_context_ids to YOUR packer (post-filter, post-rerank, post-truncate).
# Prefer LlamaIndex RetrieverEvaluator or Ragas IDBasedContextRecall when already in-stack:
# https://developers.llamaindex.ai/python/examples/evaluation/retrieval/retriever_eval/
# https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/
Canary without a full golden set
Log fusion_source per top-K hit (dense | bm25 | both). If BM25-only share flatlines after a deploy, hybrid may be silently vector-only.
Technical troubleshooting matrix
| Failure signature |
Root cause |
Diagnostic & fix |
| Empty results for one tenant |
ANN then metadata filter; fetch_k too small |
Pre-filter or raise fetch_k; verify filter field cardinality |
| Exact error codes / SKUs missed |
Dense-only retrieval |
Add BM25/sparse arm; query rewrite that keeps literals |
| Hybrid no better than dense |
Post-RRF similarity threshold drops BM25-only |
Threshold on dense arm only; keep lexical survivors |
| High similarity, wrong answer |
Chunk lacks full fact (boundary/table split) |
Structural chunking; inspect gold span vs chunk text |
| Correct doc, wrong chunk |
Boilerplate-dominated embeddings |
Strip headers/footers; smaller semantic units |
| Scores ~0 but “good” answers |
Distance/metric or embed-model mismatch |
Pin same embed_model on index build and query |
| Recall OK, users still complain |
Faithfulness / truncation of packed context |
Diff retrieved vs final_context_ids; generation eval |
| Metric improved, UX worse |
Recall↑ MRR↓ |
Add reranker; report both metrics in CI |
Best practices
- Build the eval set from real support/search queries, not only synthetic FAQ paraphrases.
- Version corpus + chunker + embedding model together; stamp
embedding_model on chunks for gradual rembeds.
- Deduplicate near-identical chunks before fusion to stop footer clones from crowding top-K.
- Keep retrieval CI separate from answer faithfulness CI (
@ai-evaluation-dataset, @ai-citation-verification).
- For evidence-dense docs, prefer structural/hierarchical chunk checks; for sparse factoid QA, simple token/sentence baselines are often enough — measure, don’t assume.
- Gate merges on Hit-rate@K + Recall@K + MRR with thresholds chosen from your baseline, not internet folklore.
Limitations
- This skill does not certify production quality for every vector DB or framework version.
- LLM-judge context metrics are proxies; disagreement with ID-based labels should be investigated.
- Multimodal/table-heavy corpora need parser-specific audits beyond text metrics.
- Stop and ask if ACL rules, index write permission, or success thresholds are missing.
Related skills
@ai-evaluation-dataset — build/version labeled query sets and gold IDs
@llm-json-contract-check — structured judge / scorer outputs
@prompt-regression-gate — prompt/packer changes with frozen retrieval
@ai-citation-verification — citation support against retrieved passages
@ai-pii-redaction-review — redact traces before sharing eval dumps
@llm-cost-latency-benchmark — cost/latency of rerank + larger K
@qdrant / @chromadb / @elasticsearch — store-specific filter and hybrid knobs
Agent Operational Directive
MANDATORY: Audit retrieval with stage isolation and metrics at production K. Log final packed context IDs. Change one variable at a time against a frozen baseline. Never recommend ACL bypasses to inflate recall. Report retrieval metrics separately from answer quality, latency, and cost.
Source anchors (research)
1---2name: rag-retrieval-audit3description: "Diagnose missing evidence in a RAG pipeline with labeled queries, chunk/trace inspection, Recall@K/MRR/nDCG, and stage-isolated failure taxonomy. Use when retrieval looks fine but answers fail, or when index/chunk/filter/rerank changes need before/after proof."4category: development5risk: safe6source: self7source_type: self8date_added: "2026-09-11"9tags: ["ai-workflows", "evaluation", "rag", "retrieval", "recall", "mrr", "ndcg", "chunking", "hybrid-search"]10tools: ["claude", "cursor", "gemini", "codex"]11metadata:12 source: https://skillmd.com/skills/alivirgo/rag-retrieval-audit13---1415# RAG Retrieval Audit AI Skill Guide (Claude)1617## Overview & Engine Architecture1819A RAG system fails in stages. Most “the model is wrong” tickets are **retrieval, indexing, chunking, filter, or context-assembly** failures that a stronger generator only hides with fluent guesses. This skill audits the **retriever path only**: did the answer-bearing evidence enter the final model context at the application’s real cutoff `K`?2021Claude operates as a Principal RAG Reliability Engineer, specializing in **stage-isolated failure taxonomy**, **ID-based Recall@K / MRR / nDCG**, **chunk-boundary forensics**, **hybrid-search arm health**, and **ACL-safe relevance denominators**.2223```24┌─────────────────────────────────────────────────────────────┐25│ RAG Retrieval Audit Stack │26│ │27│ Corpus governance │28│ ├── Source docs → parse → chunk → embed → index revision │29│ └── Metadata: ACL, tenant, version/hash, embedding_model │30│ │31│ Query path │32│ ├── Rewrite / expand → dense ± sparse (BM25) → fuse (RRF) │33│ ├── Optional rerank → filters/thresholds → context pack │34│ └── Generator (out of scope for retrieval metrics) │35│ │36│ Audit harness │37│ ├── Labeled queries + eligible relevant chunk/doc IDs │38│ ├── Per-query traces (retrieved IDs, ranks, scores, filters│39│ └── Metrics @ app K: hit-rate, Recall@K, Precision@K, MRR, │40│ nDCG; separate faithfulness later │41└─────────────────────────────────────────────────────────────┘42```4344---4546## When to use / when not to4748**Use when**4950- Answers are confident but wrong, and you need to know if evidence was missing vs ignored.51- You are changing chunk size, embeddings, hybrid weights, filters, or rerankers.52- Production has zero-result or “wrong doc family” clusters.53- Compliance requires proving which sources were eligible for a user.5455**Do not use when**5657- The task is pure prompt/style polish with a fixed golden context (use generation evals).58- You lack any labeled relevant IDs **and** refuse LLM-as-judge proxies (build labels first via `@ai-evaluation-dataset`).59- Someone asks to bypass ACLs so recall looks better — refuse; shrink the relevance denominator instead.6061---6263## Operational Capabilities & Agent Directives64651. **Freeze one baseline**: index revision, embedding model ID, chunker version, `K`, filters, fusion weights, reranker, and prompt context budget before any experiment.662. **Label at chunk (or passage) IDs**, not vague “the handbook.” Prefer `source_id + content_hash + chunk_id`.673. **Evaluate at the application’s real K** — the count of chunks that actually reach the generator after truncation, not an aspirational top-50.684. **Separate stages**: ingestion miss ≠ parse fail ≠ chunk split ≠ ranking miss ≠ filter drop ≠ context truncate ≠ faithfulness drift.695. **Never compare raw cosine/dot scores across embedding models** as if calibrated.706. **Change one variable at a time**; keep a frozen baseline run for every PR.717. **ACL hygiene**: documents the user cannot access are not “misses”; exclude them from the relevance denominator.7273---7475## Failure taxonomy (diagnose in this order)7677| Stage | Symptom | Typical root cause |78| :--- | :--- | :--- |79| **Ingestion** | Relevant `source_id` absent from index | Sync lag, failed job, wrong collection, deleted tombstone |80| **Parse** | Doc present; answer text garbled/missing | PDF/table/OCR loss; HTML boilerplate; empty extracted body |81| **Chunk** | Answer split across boundaries; half-table rows | Fixed-token splits; tiny overlap; footer pollution in embeddings |82| **Embed/index** | Near-zero or nonsense scores; empty after migration | Wrong `embedding_model` on query vs corpus; metric mismatch |83| **Retrieve** | Relevant ID never in top-K (any arm) | Weak dense match on IDs/codes; BM25 arm dead; too-small `fetch_k` |84| **Fuse/threshold** | Hybrid “works” but lexical-only hits vanish | Similarity floor applied **after** RRF to fused list |85| **Filter/ACL** | Empty or wrong-tenant hits | Post-filter after ANN (`fetch_k` then filter) vs pre-filter; over-tight metadata |86| **Rerank/pack** | ID retrieved then dropped from final prompt | Reranker demotion; duplicate collapse; token budget truncation |87| **Generate** | Correct chunks in prompt; answer still wrong | Faithfulness — **not** a retrieval fix (track separately) |8889Practitioner pattern (r/Rag): treat “RAG is inaccurate” as retrieval until proven otherwise; only then score faithfulness against the assembled context.9091---9293## Metrics — what each number means9495Use **ID-based** metrics when gold chunk/doc IDs exist (preferred). Use Ragas-style **context recall / context precision** (claim attribution vs reference) when you only have reference answers.9697| Metric | Question | Order-sensitive | Reach for it when |98| :--- | :--- | :--- | :--- |99| **Hit-rate@K** | ≥1 relevant ID in top K? | no | Cheap CI smoke |100| **Recall@K** | Fraction of gold IDs in top K | no | Missing evidence is costly |101| **Precision@K** | Fraction of top K that are relevant | no | Context budget is tight |102| **MRR** | Reciprocal rank of first relevant | yes | Generator mostly uses top 1–2 |103| **nDCG@K** | Graded ranking quality | yes | Reranker / graded labels |104| **Context recall** (Ragas) | Reference claims supportable from retrieved text | n/a | No gold chunk IDs yet |105| **Context precision** (Ragas) | Relevant chunks ranked early vs noise | yes | Noise displaces signal |106107**Diagnostic pairs**108109- Recall↓, MRR irrelevant → chunking / embedding / not in index.110- Recall steady, MRR↓ → ranking / missing reranker / fusion weights.111- Both OK, answers wrong → faithfulness or packing/truncation (log **final** prompt context).112- Recall↑ but MRR↓ after a change → more gold in the set but buried — generation often worsens.113114Do **not** gate releases only on end-to-end answer accuracy; it masks 20–40% retrieval regressions when the model bluffs.115116---117118## Audit procedure119120### 1. Inputs to collect (block if missing)121122- Bounded query set (≥30 real user questions; synthetic-only sets are optional extras).123- Per query: eligible relevant chunk IDs **under the same ACL** as production.124- Index revision / collection name, embedding model, chunker config, `K`, filters, hybrid weights, reranker ID.125- Authorization to mutate only a **test** index unless prod change is approved.126127### 2. Per-query trace schema (mandatory fields)128129Log for every audited query:130131```json132{133 "query_id": "q-1042",134 "query_text": "...",135 "user_acl": ["tenant:acme", "role:support"],136 "index_revision": "idx-2026-10-01T12:00Z",137 "embedding_model": "bge-m3",138 "k_final": 8,139 "query_rewrite": null,140 "filters_applied": {"tenant": "acme"},141 "arms": {142 "dense": [{"id": "c1", "rank": 1, "score": 0.71}],143 "bm25": [{"id": "c9", "rank": 1, "score": 12.4}]144 },145 "fused": [{"id": "c1", "rank": 1, "rrf": 0.032, "sources": ["dense"]}],146 "after_rerank": [{"id": "c9", "rank": 1, "score": 0.88}],147 "final_context_ids": ["c9", "c1", "c3"],148 "final_context_chars": 6120,149 "gold_ids": ["c9"],150 "notes": ""151}152```153154Critical: log **`final_context_ids`** (what the model saw), not only first-stage retrieval. Truncation and rerank drops are common silent failures.155156### 3. Score and classify157158For each query, mark primary failure stage from the taxonomy. Aggregate:159160- Mean Hit-rate@K, Recall@K, Precision@K, MRR, nDCG@K at **production K**.161- Slice by: zero-result, filter-excluded gold, BM25-only share of top-K, document family, language.162163### 4. Single-variable experiments164165Preserve baseline. Typical order of levers (stop when Recall@K and MRR meet gate):1661671. Confirm gold sources are indexed and parsed (spot-open chunk text).1682. Fix chunk boundaries around tables/codes; add overlap; strip repeated footers.1693. Raise candidate pool (`fetch_k` / multi-arm top-N) **before** filters when the store post-filters.1704. Prefer **pre-filter** (or filtered ANN) over fetch-then-filter when tenants are sparse in the ANN shortlist ([LangChain FAISS filter ordering](https://github.com/langchain-ai/langchain/issues/28413)).1715. Hybrid: ensure BM25-only hits are not killed by a post-fusion vector similarity floor ([r/Rag hybrid bug reports](https://www.reddit.com/r/Rag/comments/1uvl9cx/)).1726. Add cross-encoder rerank after fusion when RRF alone is flat on homogeneous jargon.1737. Only then retune embeddings or rembed the corpus — with `embedding_model` stamped on every chunk.174175### 5. Deliverable176177Ship:1781791. Per-query failure table (query_id, stage, gold_ids, final_context_ids, hit, ranks).1802. Aggregate metrics before/after with identical `K` and ACL rules.1813. Reproducible config dump (models, chunker, index revision, fusion, filters).1824. Explicit “not retrieval” bucket for faithfulness follow-ups.1835. ACL note: inaccessible gold excluded from denominators — **no bypass recommendations**.184185---186187## Production example: ID-based retrieval eval harness188189```python190"""Minimal RAG retrieval audit: Recall@K, Precision@K, MRR, hit-rate."""191from __future__ import annotations192193from dataclasses import dataclass194from typing import Callable, Sequence195196197@dataclass(frozen=True)198class Case:199 query_id: str200 query: str201 gold_ids: frozenset[str] # eligible under the same ACL202203204def recall_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:205 if not gold:206 return float("nan") # exclude ACL-empty cases from means207 top = list(retrieved)[:k]208 return len(gold.intersection(top)) / len(gold)209210211def precision_at_k(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:212 top = list(retrieved)[:k]213 if not top:214 return 0.0215 return len(gold.intersection(top)) / len(top)216217218def mrr(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:219 for i, doc_id in enumerate(list(retrieved)[:k], start=1):220 if doc_id in gold:221 return 1.0 / i222 return 0.0223224225def hit_rate(retrieved: Sequence[str], gold: frozenset[str], k: int) -> float:226 return 1.0 if gold.intersection(list(retrieved)[:k]) else 0.0227228229def audit(230 cases: Sequence[Case],231 retrieve_final_context_ids: Callable[[str], Sequence[str]],232 k: int,233) -> list[dict]:234 rows = []235 for case in cases:236 ids = list(retrieve_final_context_ids(case.query))237 rows.append(238 {239 "query_id": case.query_id,240 "hit": hit_rate(ids, case.gold_ids, k),241 "recall": recall_at_k(ids, case.gold_ids, k),242 "precision": precision_at_k(ids, case.gold_ids, k),243 "mrr": mrr(ids, case.gold_ids, k),244 "retrieved": ids[:k],245 "gold": sorted(case.gold_ids),246 }247 )248 return rows249250251# Wire retrieve_final_context_ids to YOUR packer (post-filter, post-rerank, post-truncate).252# Prefer LlamaIndex RetrieverEvaluator or Ragas IDBasedContextRecall when already in-stack:253# https://developers.llamaindex.ai/python/examples/evaluation/retrieval/retriever_eval/254# https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/255```256257### Canary without a full golden set258259Log `fusion_source` per top-K hit (`dense` | `bm25` | `both`). If BM25-only share flatlines after a deploy, hybrid may be silently vector-only.260261---262263## Technical troubleshooting matrix264265| Failure signature | Root cause | Diagnostic & fix |266| :--- | :--- | :--- |267| Empty results for one tenant | ANN then metadata filter; `fetch_k` too small | Pre-filter or raise `fetch_k`; verify filter field cardinality |268| Exact error codes / SKUs missed | Dense-only retrieval | Add BM25/sparse arm; query rewrite that keeps literals |269| Hybrid no better than dense | Post-RRF similarity threshold drops BM25-only | Threshold on dense arm only; keep lexical survivors |270| High similarity, wrong answer | Chunk lacks full fact (boundary/table split) | Structural chunking; inspect gold span vs chunk text |271| Correct doc, wrong chunk | Boilerplate-dominated embeddings | Strip headers/footers; smaller semantic units |272| Scores ~0 but “good” answers | Distance/metric or embed-model mismatch | Pin same `embed_model` on index build and query |273| Recall OK, users still complain | Faithfulness / truncation of packed context | Diff `retrieved` vs `final_context_ids`; generation eval |274| Metric improved, UX worse | Recall↑ MRR↓ | Add reranker; report both metrics in CI |275276---277278## Best practices2792801. Build the eval set from **real** support/search queries, not only synthetic FAQ paraphrases.2812. Version corpus + chunker + embedding model together; stamp `embedding_model` on chunks for gradual rembeds.2823. Deduplicate near-identical chunks before fusion to stop footer clones from crowding top-K.2834. Keep retrieval CI separate from answer faithfulness CI (`@ai-evaluation-dataset`, `@ai-citation-verification`).2845. For evidence-dense docs, prefer structural/hierarchical chunk checks; for sparse factoid QA, simple token/sentence baselines are often enough — measure, don’t assume.2856. Gate merges on Hit-rate@K + Recall@K + MRR with thresholds chosen from your baseline, not internet folklore.286287---288289## Limitations290291- This skill does not certify production quality for every vector DB or framework version.292- LLM-judge context metrics are proxies; disagreement with ID-based labels should be investigated.293- Multimodal/table-heavy corpora need parser-specific audits beyond text metrics.294- Stop and ask if ACL rules, index write permission, or success thresholds are missing.295296---297298## Related skills299300- `@ai-evaluation-dataset` — build/version labeled query sets and gold IDs301- `@llm-json-contract-check` — structured judge / scorer outputs302- `@prompt-regression-gate` — prompt/packer changes with frozen retrieval303- `@ai-citation-verification` — citation support against retrieved passages304- `@ai-pii-redaction-review` — redact traces before sharing eval dumps305- `@llm-cost-latency-benchmark` — cost/latency of rerank + larger K306- `@qdrant` / `@chromadb` / `@elasticsearch` — store-specific filter and hybrid knobs307308---309310## Agent Operational Directive311312> **MANDATORY**: Audit retrieval with stage isolation and metrics at production `K`. Log final packed context IDs. Change one variable at a time against a frozen baseline. Never recommend ACL bypasses to inflate recall. Report retrieval metrics separately from answer quality, latency, and cost.313314---315316## Source anchors (research)317318- [Ragas Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/) / [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)319- [LlamaIndex RetrieverEvaluator](https://developers.llamaindex.ai/python/examples/evaluation/retrieval/retriever_eval/)320- [LangChain FAISS filter-after-ANN empty results](https://github.com/langchain-ai/langchain/issues/28413)321- [LlamaIndex zero/near-zero scores / embed mismatch](https://github.com/run-llama/llama_index/issues/18890)322- r/Rag: retrieval-vs-generation split; chunk boundary failures; post-RRF threshold killing BM25