RAG Pipeline Architect

RAG Pipeline Architect

Design production-grade RAG pipelines from document ingestion to retrieval-augmented response generation, with systematic evaluation and optimization at every stage.

When to Use

• Designing a new RAG system from scratch
• Diagnosing "the LLM doesn't use my documents" problems
• Choosing between chunking strategies, embedding models, or vector databases
• Optimizing retrieval quality (precision, recall, relevance)
• Migrating from naive RAG to advanced RAG patterns
• Building evaluation harnesses for retrieval quality

RAG Architecture Tiers

Tier 1: Naive RAG

The simplest end-to-end pipeline. Good for prototyping.

Documents → Chunk → Embed → Store → Query → Retrieve → Generate

Pros: Fast to build, easy to understand Cons: Poor handling of complex queries, no re-ranking, chunk boundary issues

Tier 2: Advanced RAG

Adds pre-retrieval and post-retrieval optimization stages.

Documents → Clean → Chunk (smart) → Embed → Store
                                                ↓
Query → Rewrite → Embed → Retrieve → Re-rank → Filter → Generate
  ↑                                                        ↓
  └──────────── Self-reflection / Retry ──────────────────┘

Key additions:

• Query rewriting and expansion
• Hybrid search (dense + sparse)
• Re-ranking with cross-encoders
• Self-reflection loop for answer validation

Tier 3: Modular RAG

Fully composable pipeline with pluggable components.

┌─── Ingestion Pipeline ────────────────────────────┐
│  Source Connectors → Parsing → Cleaning →          │
│  Chunking → Enrichment → Embedding → Indexing      │
└────────────────────────────────────────────────────┘

┌─── Query Pipeline ────────────────────────────────┐
│  Query Analysis → Intent Classification →          │
│  Query Routing → Retrieval Strategy Selection →    │
│  Multi-source Retrieval → Fusion → Re-ranking →   │
│  Context Assembly → Generation → Validation        │
└────────────────────────────────────────────────────┘

Workflow

Phase 1: Document Analysis & Ingestion Design

1. Audit the corpus:

   • Document types (PDF, HTML, Markdown, code, tables, images)
   • Total size and document count
   • Update frequency (static vs. streaming)
   • Language distribution
   • Structural complexity (headings, tables, nested lists, code blocks)

2. Choose a parsing strategy:

   Document Type	Recommended Parser	Notes
   PDF (text)	PyMuPDF / pdfplumber	Preserves layout and tables
   PDF (scanned)	Tesseract + layout detection	Add OCR preprocessing
   HTML	BeautifulSoup + Readability	Strip boilerplate first
   Markdown	Native parsing	Preserve heading hierarchy
   Code files	Tree-sitter	Parse by function/class boundaries
   Spreadsheets	pandas	Convert to structured text with headers
   Images/diagrams	Vision LLM (GPT-4V, Claude)	Generate text descriptions

3. Design the cleaning pipeline:

   • Remove boilerplate (headers, footers, navigation)
   • Normalize whitespace and encoding
   • Extract and preserve metadata (title, author, date, source URL)
   • Handle special characters and formatting artifacts

Phase 2: Chunking Strategy

Choose the chunking approach based on document structure:

Chunking decision tree:

Is the document well-structured (headings, sections)?
├── Yes → Document-structure chunking
│         + Parent-child indexing for hierarchy
├── No → Is it source code?
│        ├── Yes → Code-aware chunking (tree-sitter)
│        └── No → Is high precision critical?
│                 ├── Yes → Sentence-window chunking
│                 └── No → Recursive character chunking

Critical rules:

• Always include metadata in chunks (source, page, section title)
• Never break mid-sentence or mid-code-block
• Test chunk quality by reading 20 random chunks — each should be self-contained and meaningful

Phase 3: Embedding & Indexing

1. Choose embedding model:

   Model	Dimensions	Max Tokens	Strengths
   OpenAI text-embedding-3-large	3072	8191	Best general quality
   OpenAI text-embedding-3-small	1536	8191	Cost-effective
   Cohere embed-v4	1024	512	Multilingual excellence
   Voyage AI voyage-3-large	1024	32000	Long-context, code
   BGE-M3 (open source)	1024	8192	Multilingual, free
   Jina embeddings-v3	1024	8192	Multilingual, open
   GTE-Qwen2 (open source)	1024	8192	Chinese + English, free

2. Choose vector database:

   Database	Type	Best For	Scale
   Pinecone	Managed	Production, zero-ops	Billions
   Weaviate	Managed/Self	Hybrid search, multi-modal	Millions-Billions
   Qdrant	Self-hosted	Performance, filtering	Millions
   ChromaDB	Embedded	Prototyping, local dev	Thousands-Millions
   pgvector	Postgres extension	Existing Postgres stack	Millions
   FAISS	In-memory library	Research, benchmarking	Millions
   Milvus	Distributed	Large-scale production	Billions

3. Indexing best practices:

   • Store both the embedding vector AND the original text
   • Include rich metadata for filtering (source, date, category, language)
   • Build separate indexes for different document types if needed
   • Implement incremental indexing for updates (don't re-index everything)

Phase 4: Retrieval Strategy

Design the query-time retrieval pipeline:

1. Query preprocessing:

   • Detect query intent (factual, analytical, procedural, conversational)
   • Expand query with synonyms or related terms (query expansion)
   • Rewrite ambiguous queries using the LLM (HyDE: Hypothetical Document Embeddings)
   • Decompose complex queries into sub-queries

2. Retrieval methods (choose one or combine):

   Method	How It Works	When to Use
   Dense retrieval	Embedding similarity (cosine/dot product)	Semantic meaning matches
   Sparse retrieval	BM25 / TF-IDF keyword matching	Exact term matches, names, codes
   Hybrid	Dense + Sparse with score fusion	Best overall quality
   Multi-query	Generate N query variants, retrieve for each, merge	Complex or ambiguous queries
   Parent-child	Retrieve small chunks, return parent chunks	Need both precision and context

3. Re-ranking:

   • Apply a cross-encoder re-ranker (e.g., Cohere Rerank, BGE-Reranker)
   • Re-ranking is the single highest-impact improvement for most RAG systems
   • Retrieve more candidates than needed (e.g., top-20), re-rank to top-5

4. Post-retrieval filtering:

   • Remove chunks below a relevance score threshold
   • Deduplicate near-identical chunks
   • Ensure diversity (don't return 5 chunks from the same page)
   • Respect recency requirements (filter by date if needed)

Phase 5: Context Assembly & Generation

1. Assemble the context:

   • Order chunks by relevance (most relevant first AND last — avoid "lost in the middle")
   • Include source attribution metadata
   • Add a preamble: "Answer based ONLY on the following context. If the context doesn't contain the answer, say so."

2. Generation prompt template:

   You are a [domain] expert. Answer the user's question based on the
   provided context.
   
   ## Rules
   - Only use information from the provided context
   - Cite sources using [Source: document_name, page X]
   - If the context is insufficient, explicitly state what's missing
   - Never fabricate information
   
   ## Context
   {retrieved_chunks_with_metadata}
   
   ## Question
   {user_query}
   
   ## Answer
   

3. Response validation:

   • Check for hallucination: Does every claim have a supporting chunk?
   • Check for completeness: Did the response address all parts of the query?
   • Check for citation accuracy: Do citations point to the right chunks?

Phase 6: Evaluation

Evaluate the RAG pipeline at two levels:

Retrieval evaluation:

End-to-end evaluation:

Build an eval dataset:

• Minimum 50 question-answer pairs with annotated relevant documents
• Include: easy factual, multi-hop reasoning, unanswerable, and adversarial queries
• Version control the eval set and never train/tune on it

Common Failure Modes & Fixes

Guidelines

• Start simple, iterate: Begin with Tier 1 (naive RAG), measure, then add complexity only where metrics show gaps.
• Re-ranking is the best single upgrade: If you do one thing to improve quality, add a cross-encoder re-ranker.
• Chunk quality > chunk quantity: 5 excellent chunks beat 20 mediocre ones.
• Always evaluate: Without metrics, you're guessing. Build the eval harness before optimizing.
• Version everything: Embedding models, chunk strategies, and prompts should all be versioned and traceable.