RAG Architecture

RAG architecture

Verified 2026-07-29. Model names, prices and benchmark positions in this space go stale in months — treat every named model below as an example of a class, not a recommendation to copy.

1. When RAG fails, it is retrieval that failed

Industry analysis through 2026 consistently puts the failure point at retrieval roughly three quarters of the time, not generation. The visible symptom is a bad answer, so the instinct is to edit the prompt — and prompt edits cannot recover a chunk that never entered the context.

Before touching the prompt, ask: was the right chunk retrieved at all? Log the retrieved chunk ids for every query. That single log line separates "the model ignored good context" from "the model never saw it", and those have opposite fixes.

2. RAG vs CAG vs long context — a routing decision, not a religion

Cache-augmented generation (CAG) loads the whole corpus into a cached prompt once and answers every query against it, with no retrieval step. Long-context models priced with prompt caching made this viable, and it is a peer architecture to RAG, not a successor.

Decide on the axes, per workload:

The production answer is usually both: a router sending the hot path to the cache and the cold path to retrieval. Score both with the same rubric or the comparison is decided by whichever you instrumented better.

Teams that declare "RAG is dead" ship the wrong architecture for half their workloads and back-port RAG within a quarter. Per-tenant data and citation requirements are the two constraints that most reliably force retrieval, and neither is negotiable by tuning.

3. Chunking — where pipelines fail silently

The test for a chunk: can it answer a question on its own? A chunk that begins "This means the threshold is exceeded" is unretrievable, because the query that needs it does not look like it.

Carry the heading path into the chunk text. Contract > Schedule 2 > Payment terms costs a few tokens and makes an otherwise ambiguous fragment both retrievable and readable.

Parent–child is the pattern most worth adopting early. Small chunks retrieve precisely; large chunks answer completely. Storing the small one for the vector and returning its parent for the context gets both, and removes most of the chunk-size argument.

Semantic chunking is not free — it embeds sentence-by-sentence at ingest. Justify it against structural chunking on your own data before paying for it everywhere.

4. Hybrid retrieval is the default

Dense vectors alone fail on exactly the queries users care most about: product codes, error numbers, invoice references, surnames, rare acronyms. Embeddings encode meaning, and an identifier has no meaning to encode — ERR-4471 and ERR-4417 sit almost on top of each other.

The 2026 default is BM25 (lexical) + dense vectors, fused — commonly by reciprocal rank fusion — with a graph or structured layer added only where entity relationships genuinely drive the questions.

Add the graph layer on evidence, not fashion. GraphRAG earns its substantial ingest cost on multi-hop and relationship questions ("who signed the contracts that reference this clause"), and adds cost with no benefit on lookup questions.

Query rewriting before retrieval — expanding a terse or pronoun-laden follow-up into a standalone query — is one of the cheapest real wins, particularly in multi-turn chat where the user's literal words retrieve nothing.

5. Reranking

A cross-encoder reranker scores query and chunk together, which a bi-encoder cannot do, and it is consistently one of the largest single contributors to final ranking quality.

Retrieve ~20, rerank, send 3–5. Retrieving wide and cutting hard beats retrieving narrow: recall is what you cannot recover later, and precision is what the reranker fixes.

More context is not better. A window padded with near-misses diffuses attention and inflates cost. There is a real quality peak, and it is lower than most first implementations assume.

6. Embedding models

Public benchmark position is a shortlist, never a decision. Benchmarks get overfitted, and your corpus is not the benchmark. Build a golden set of 50–100 real queries with known-correct chunks and measure recall@k on your own data — it takes an afternoon and outranks every leaderboard.

Changing the embedding model means re-embedding the entire corpus. Vectors from two models are not comparable, and mixing them produces retrieval that is quietly, unfixably wrong rather than broken. Plan the migration — dual-write, backfill, cut over — before the first model choice, and store the model id and version alongside every vector so a mixed store is detectable.

7. Vector store

Default to the database you already run. A separate vector store is a second system to secure, back up, monitor and keep consistent, and it earns that only at scale or under a workload the extension genuinely cannot serve.

Metadata filtering is the feature that decides this in practice, not raw query speed. Almost every real system filters — by tenant, date, document type, permission — and engines differ enormously in whether filters are applied during the vector search or as a post-filter that silently returns fewer than k results.

Store text alongside vectors, or store a durable pointer. Rebuilding the answer from ids against a source that has since changed is a slow-motion correctness bug.

8. Access control — filter during retrieval, never after

Permissions belong in the query, as chunk-level metadata. Retrieving broadly and filtering the results afterwards means the model has already seen material the user may not access, and any summary, citation or follow-up can leak it. Post-filtering also breaks k silently.

A document's permissions must be carried onto every chunk derived from it, and re-derived when the source's permissions change. This is the most commonly skipped step when a prototype meets its first real tenant.

9. Evaluation — measure retrieval and generation separately

Recall is the higher-leverage retrieval metric: if the right chunk never enters the candidate set, no reranking, prompting or model upgrade recovers it. Precision is a cost and attention problem; recall is a correctness ceiling.

Faithfulness is the ship gate. Widely-cited practice puts the production target around 90% and treats sub-70% as unsafe to ship. Pick your own threshold, but pick it before you see the score.

A golden set beats a framework. Evaluation tooling helps, but a stored set of real queries with known-correct chunks and expected answers is what makes a change reviewable. Without it, every tuning round is a vibe.

For agentic RAG, score the trajectory, not just the final answer — how many retrievals, whether they were redundant, whether the agent stopped when it had enough. Faithfulness is necessary and not sufficient once the system acts on what it retrieved.

10. Ingestion and freshness

• Incremental, content-hashed ingestion. Re-embedding an unchanged document is pure cost.
• Deletes must propagate. An orphaned vector for a deleted document is a confident citation of something that no longer exists — and in a regulated setting, a retention breach.
• Version the chunking scheme, not just the data. A chunker change invalidates the corpus exactly as an embedding-model change does.
• Retrieved content is untrusted input. A document can carry instructions aimed at your model — see llm-security for the injection and poisoning defences that belong at this boundary.

11. Anti-patterns

• Tuning the prompt when retrieval is what failed. Log the retrieved chunk ids first.
• Declaring RAG obsolete because long context got cheap — then rediscovering per-tenant data and citations.
• Dense-only retrieval on a corpus full of identifiers, codes and names.
• Chunks that cannot stand alone, stripped of their heading context.
• Stuffing the window with 20 chunks because they fit.
• Choosing an embedding model from a leaderboard without a golden set.
• Mixing vectors from two embedding models in one index.
• Post-filtering for permissions after the model has seen the content.
• Running a separate vector store when the existing database would do.
• No evaluation set, so every change is judged on the last query someone happened to try.
• Trusting retrieved text as though the corpus were part of your prompt.