RAG Patterns

RAG (Retrieval-Augmented Generation) Patterns

Overview

RAG augments large language models with real-time external information retrieval, addressing a fundamental limitation: LLMs only know what they were trained on and hallucinate when uncertain. By retrieving relevant documents from vector databases before generation, RAG provides up-to-date, domain-specific context that dramatically improves accuracy. Organizations implementing RAG report 78% improvement in response accuracy for domain-specific queries vs. vanilla LLMs. As of 2025, 63% of enterprise AI projects incorporate retrieval augmentation. The architecture: chunk documents → generate embeddings → index in vector store → retrieve relevant chunks for each query → augment prompt with context → generate response.

When to Use

• Building AI assistants that need current information beyond LLM training cutoff
• Reducing hallucinations in high-stakes domains (medical, legal, financial)
• Creating internal knowledge base chatbots for companies (docs, policies, support)
• Implementing semantic search with generative summarization
• Personalizing LLM responses with user-specific data or context
• Building AI applications where citations/sources are required
• Scaling knowledge systems without retraining expensive foundation models

The Process

Step 1: Prepare and Chunk Documents

Break source documents into chunks (typically 500-1000 tokens). Use structure-aware segmentation respecting paragraphs, sections, or semantic boundaries. Implement maximum/minimum chunk size constraints. Preserve metadata (source, timestamp, author). Example: 100-page technical manual → 200 chunks of ~500 tokens each, preserving section headers as metadata for context.

Step 2: Generate Embeddings and Index

Convert each chunk to vector embedding using embedding model (OpenAI ada-002, Voyage, Cohere). Store vectors in specialized database (Pinecone, Weaviate, Qdrant, ChromaDB). Index for fast similarity search. Update regularly as documents change. Example: Use text-embedding-3-small (384 dimensions) for speed or text-embedding-3-large (3072 dimensions) for accuracy, store in Pinecone with metadata filters.

Step 3: Implement Retrieval Strategy

On user query, generate query embedding using same model. Retrieve top-k most similar chunks (typically k=3-10) via cosine similarity. Optionally rerank results using cross-encoder for accuracy. Consider hybrid search (semantic + keyword). Example: User asks "What's our PTO policy?" → embed query → retrieve 5 most similar chunks from HR handbook → rerank by relevance.

Step 4: Augment Prompt with Retrieved Context

Construct prompt: system instructions + retrieved chunks + user query. Clearly delimit context (XML tags or markdown sections). Include source citations. Instruct model to answer based on provided context only. Example: <context>Chunk 1: ...\\nChunk 2: ...</context> <question>User query</question> Instructions: Answer using only the context provided. Cite sources.

Step 5: Generate Response and Cite Sources

Pass augmented prompt to LLM (GPT-4, Claude, Gemini). Model generates answer grounded in retrieved context. Include source citations (chunk IDs, page numbers, document names). Optionally implement confidence scoring. Example: Response: "According to the Employee Handbook (Section 3.2), full-time employees accrue 15 PTO days annually..."

Step 6: Implement Advanced Patterns (Optional)

Choose architecture based on complexity: (a) Simple RAG - basic retrieval, (b) RAG with Memory - conversational context across turns, (c) Long RAG - process entire documents not just chunks, (d) Self-RAG - model decides when to retrieve and self-critiques, (e) Agentic RAG - multi-step reasoning with iterative retrieval. Example: For customer support, use RAG with Memory to maintain conversation context across multi-turn dialog.

Step 7: Monitor, Evaluate, and Iterate

Track metrics: retrieval precision/recall (are right chunks retrieved?), answer accuracy (is final response correct?), citation quality (sources match claims?), latency (response time). Fine-tune chunking strategy, embedding model, retrieval parameters, and prompt based on performance. Example: If users report missing information, increase k from 5 to 10 chunks or switch to hybrid semantic+keyword retrieval.

Example Application

Situation: Legal tech startup building AI contract analyzer. Initial GPT-4 implementation hallucinated legal precedents and provided outdated advice. Needed to ground responses in current case law and firm's proprietary templates.

Application:

• Step 1: Chunked 5,000 legal documents (contracts, case law, internal memos) into 50,000 chunks of ~600 tokens, preserving document type and date metadata
• Step 2: Generated embeddings with Cohere embed-english-v3.0, indexed in Weaviate with metadata filters (jurisdiction, practice area, date)
• Step 3: Implemented hybrid retrieval: semantic search (top 10 chunks) + keyword search for legal citations + reranking with cross-encoder → final top 5 chunks
• Step 4: Prompt structure: "You are a legal analyst. Use only the provided context to answer. Cite specific clauses and cases. If unsure, say so."
• Step 5: Claude 4 generated responses with inline citations: "[Source: Contract Template v3.2, Clause 8.1]"
• Step 6: Used Long RAG pattern for full contract analysis (processing entire 50-page contracts as context)
• Step 7: Monitored retrieval quality - discovered date filtering improved accuracy, switched to weekly index updates for case law

Outcome: Hallucination rate dropped from 23% to 3%, lawyer trust increased (89% approval rating), response accuracy validated against manual review reached 94%, average contract analysis time reduced from 4 hours to 15 minutes.

Anti-Patterns

• Chunks too large (>2000 tokens, lose semantic precision) or too small (<200 tokens, lose context)
• Using same embedding model for indexing and different one for querying (incompatible vector spaces)
• Retrieving too few chunks (k=1-2, missing relevant context) or too many (k=50+, overwhelming prompt)
• Not updating index regularly (stale information defeats real-time purpose)
• Failing to cite sources (users can't verify claims, defeats transparency benefit)
• Ignoring retrieval quality metrics (assuming retrieved chunks are always relevant)
• Overcomplicating architecture prematurely (start with Simple RAG, add complexity only if needed)
• Not filtering by metadata when applicable (searching all documents when user needs specific domain)

Related

• Vector Databases (Pinecone, Weaviate, Qdrant) - infrastructure for embedding storage
• Embedding Models (OpenAI, Cohere, Voyage) - convert text to vector representations
• Prompt Engineering - crafting effective prompts with retrieved context
• Fine-Tuning vs. RAG - trade-offs between approaches (RAG faster to update, fine-tuning better for style)
• LangChain / LlamaIndex - frameworks simplifying RAG implementation
• Semantic Search - retrieval technique underlying RAG
• Knowledge Graphs - alternative/complementary approach to vector retrieval
• Agentic Workflows - multi-step reasoning patterns incorporating RAG