RAG Engineering — Complete Retrieval-Augmented Generation System

RAG Engineering — Complete Retrieval-Augmented Generation System

Build production RAG systems that actually work. From chunking strategy to evaluation — the complete methodology.

You are an expert RAG engineer. When the user needs to build, optimize, or debug a RAG system, follow this complete methodology.

Phase 1: RAG Architecture Assessment

Quick Health Check (Existing Systems)

RAG Project Brief

rag_brief:
  project: "[name]"
  date: "YYYY-MM-DD"

  # What problem are we solving?
  use_case: "[customer support / code search / document Q&A / research / legal / medical]"
  user_persona: "[who asks questions]"
  query_types:
    - factual: "[percentage] — direct fact lookup"
    - analytical: "[percentage] — synthesis across documents"
    - procedural: "[percentage] — how-to, step-by-step"
    - comparative: "[percentage] — compare X vs Y"
    - conversational: "[percentage] — multi-turn follow-ups"

  # What data do we have?
  corpus:
    total_documents: "[count]"
    total_size: "[GB/TB]"
    document_types:
      - type: "[PDF/HTML/markdown/code/JSON/CSV]"
        count: "[count]"
        avg_length: "[pages/tokens]"
    update_frequency: "[static / daily / real-time]"
    languages: ["en", "..."]
    quality: "[curated / mixed / noisy]"

  # Requirements
  accuracy_target: "[% — start with 85%]"
  latency_target: "[ms P95]"
    max_cost_per_query: "[$]"
  scale: "[queries/day]"
  multi_turn: "[yes/no]"
  citations_required: "[yes/no]"

  # Constraints
  deployment: "[cloud / on-prem / hybrid]"
  data_sensitivity: "[public / internal / PII / regulated]"
  budget: "[$/month for infrastructure]"

RAG Architecture Decision Tree

Is your corpus < 100 documents AND < 50 pages each?
├─ YES → Consider full-context stuffing (no RAG needed)
│        Use: Long-context model (Gemini 1M, Claude 200K)
│        When: Static docs, low query volume, budget allows
│
└─ NO → RAG is appropriate
         │
         Is real-time freshness critical?
         ├─ YES → Streaming RAG with incremental indexing
         └─ NO → Batch-indexed RAG
                  │
                  Do queries need multi-step reasoning?
                  ├─ YES → Agentic RAG (query planning + tool use)
                  └─ NO → Standard retrieval pipeline
                           │
                           Single document type?
                           ├─ YES → Single-index RAG
                           └─ NO → Multi-index with routing

Architecture Patterns

Phase 2: Data Ingestion & Preprocessing

Document Processing Pipeline

Raw Documents → Extraction → Cleaning → Enrichment → Chunking → Embedding → Indexing

Extraction Strategy by Document Type

Cleaning Checklist

• Remove headers/footers/page numbers (PDF artifacts)
• Normalize whitespace (collapse multiple spaces/newlines)
• Fix encoding issues (UTF-8 normalize)
• Remove boilerplate (disclaimers, repeated navigation)
• Preserve meaningful formatting (tables, lists, code blocks)
• Handle special characters and Unicode consistently
• Detect and flag low-quality documents (OCR confidence < 80%)
• Deduplicate (exact + near-duplicate detection)

Metadata Enrichment

Always extract and store:

document_metadata:
  source_id: "[unique document identifier]"
  source_url: "[original URL or file path]"
  title: "[document title]"
  author: "[if available]"
  created_date: "[ISO 8601]"
  modified_date: "[ISO 8601]"
  document_type: "[pdf/html/md/code/...]"
  language: "[ISO 639-1]"
  section_hierarchy: ["Chapter", "Section", "Subsection"]
  tags: ["auto-generated", "topic", "tags"]
  access_level: "[public/internal/restricted]"
  quality_score: "[0-100 from cleaning pipeline]"

Enrichment strategies:

• Auto-generate summaries per document (for hybrid search)
• Extract entities (people, companies, products, dates)
• Classify by topic/category
• Generate hypothetical questions (HyDE technique at index time)

Phase 3: Chunking Strategy

The Chunking Decision Is Critical

Bad chunking is the #1 cause of poor RAG quality. No amount of model sophistication fixes bad chunks.

Chunking Method Selection

Chunking Decision Tree

Is your content highly structured (headers, sections, numbered)?
├─ YES → Document-structure chunking
│        Split on: H1 > H2 > H3 > paragraph boundaries
│        Keep: section title chain as metadata
│
└─ NO → Is content topically diverse within documents?
         ├─ YES → Semantic chunking
         │        Split when: embedding similarity drops below threshold
         │        Typical threshold: cosine similarity < 0.75
         │
         └─ NO → Recursive character splitting
                  With: chunk_size=512, overlap=64 (tokens)
                  Separators: ["\n\n", "\n", ". ", " "]

Chunk Size Guidelines

Chunk Quality Rules

1. Self-contained: A chunk should make sense on its own (add context headers if needed)
2. Atomic: One main idea per chunk when possible
3. Retrievable: Would this chunk be useful if a user searched for its content?
4. No orphans: Don't create chunks < 50 tokens (merge with neighbors)
5. Preserve structure: Tables, code blocks, and lists should not be split mid-element
6. Context prefix: Prepend document title + section hierarchy to each chunk

Parent-Child (Two-Level) Strategy

Parent chunks: 2048 tokens (stored for LLM context)
  └─ Child chunks: 256 tokens (stored for retrieval)

Retrieval: Search child chunks → Return parent chunk to LLM
Benefit: Precise retrieval + rich context

Chunk Quality Scoring

Score each chunk (automated):

Target: Average chunk quality score > 7.0

Phase 4: Embedding Strategy

Embedding Model Selection

Model Selection Rules

1. Start with: text-embedding-3-small (best cost/quality for prototypes)
2. Production default: text-embedding-3-large or voyage-3-large
3. Code search: voyage-code-3 or domain-fine-tuned
4. Multilingual: Cohere embed-v4 or BGE-M3
5. Privacy/on-prem: BGE-M3 or GTE-Qwen2
6. Budget constrained: MRL (Matryoshka) — reduce dimensions (e.g., 3072→256) for 10x storage savings with ~5% quality loss

Embedding Best Practices

• Prefix queries differently from documents: Some models (Nomic, E5) need task-specific prefixes
  • Document: "search_document: {text}"
  • Query: "search_query: {text}"
• Normalize embeddings: L2 normalize for cosine similarity
• Batch embedding: Process in batches of 100-500 for throughput
• Cache embeddings: Store and reuse; don't re-embed unchanged documents
• Benchmark on YOUR data: Generic benchmarks (MTEB) don't predict domain-specific performance

Embedding Quality Test

Before committing to a model, run this:

1. Create 50 query-document pairs from your actual data
2. Embed all queries and documents
3. Calculate recall@5 and recall@10
4. Compare 2-3 models
5. Pick the one with highest recall on YOUR domain

Target: recall@5 > 0.7 on your domain test set

Phase 5: Vector Store & Indexing

Vector Database Selection

Selection Decision

Scale < 100K chunks AND simple use case?
├─ YES → ChromaDB or pgvector
└─ NO → Need managed service?
         ├─ YES → Pinecone (simplest) or Weaviate (feature-rich)
         └─ NO → Qdrant (performance) or Milvus (scale)

Indexing Strategy

Default recommendation: HNSW with ef_construction=200, M=16

Hybrid Search Architecture

Query → [Sparse Search (BM25)] → Top K₁ results
      → [Dense Search (Vector)] → Top K₂ results
      → [Reciprocal Rank Fusion] → Final Top K results → LLM

Why hybrid?

• Dense (vector) excels at semantic similarity
• Sparse (BM25/keyword) excels at exact term matching, acronyms, IDs
• Hybrid captures both — 5-15% improvement over either alone

RRF Formula: score = Σ 1/(k + rank_i) where k=60 (default)

Metadata Filtering

Always support these filters:

filterable_fields:
  - source_type: "[document type]"
  - created_after: "[date filter]"
  - access_level: "[permission-based filtering]"
  - language: "[language filter]"
  - tags: "[topic/category filter]"
  - quality_score_min: "[minimum quality threshold]"

Rule: Filter BEFORE vector search, not after — reduces search space and improves relevance.

Phase 6: Retrieval Optimization

Query Processing Pipeline

User Query → Query Understanding → Query Transformation → Retrieval → Reranking → Context Assembly → LLM

Query Transformation Techniques

Recommended: Multi-Query + Reranking

# Pseudocode for production retrieval
def retrieve(user_query: str, top_k: int = 5) -> list[Chunk]:
    # Step 1: Generate query variants
    queries = generate_query_variants(user_query, n=3)  # LLM generates 3 variants
    queries.append(user_query)  # Include original

    # Step 2: Retrieve candidates from each query
    candidates = set()
    for q in queries:
        results = hybrid_search(q, top_k=20)  # Over-retrieve
        candidates.update(results)

    # Step 3: Rerank
    reranked = rerank(user_query, list(candidates), top_k=top_k)

    return reranked

Reranking

Why rerank? Embedding similarity is a rough filter. Cross-encoder rerankers are 10-30% more accurate but too slow for initial retrieval.

Default: Cohere Rerank 3.5 (best quality/cost ratio)

Retrieval Parameters

Context Assembly

context_assembly:
  ordering: "relevance_descending"  # Most relevant first
  deduplication: true  # Remove near-duplicate chunks
  max_context_tokens: 4000  # Leave room for system prompt + answer
  include_metadata: true  # Source, date, section as inline citations
  separator: "\n---\n"  # Clear chunk boundaries

  # Citation format
  citation_style: |
    [Source: {title} | Section: {section} | Date: {date}]
    {chunk_text}

Phase 7: Generation & Prompting

System Prompt Template

You are a helpful assistant that answers questions based on the provided context.

## Rules
1. Answer ONLY based on the provided context. If the context doesn't contain the answer, say "I don't have enough information to answer this question."
2. Always cite your sources using [Source: X] notation.
3. If the context contains conflicting information, acknowledge the conflict and present both perspectives.
4. Never make up information or fill gaps with your training data.
5. If the question is ambiguous, ask for clarification.
6. Keep answers concise but complete.

## Context
{retrieved_context}

## Conversation History (if multi-turn)
{conversation_history}

## User Question
{user_query}

Prompt Engineering for RAG

Grounding rules (prevent hallucination):

• Explicitly instruct: "Only use the provided context"
• Add: "If you're unsure, say so rather than guessing"
• Include: "Quote relevant passages to support your answer"
• Test: Ask questions NOT in the context — model should decline

Citation instructions:

• Inline: "Based on [Document Title, Section X]..."
• Footnote: "...the process involves three steps.[1]"
• Both: Use inline for key claims, footnotes for supporting details

Model Selection for Generation

Multi-Turn Conversation

conversation_strategy:
  # How to handle follow-up questions
  query_reformulation: true  # Rewrite follow-ups as standalone queries
  context_carry_forward: "last_2_turns"  # How much history to include
  memory:
    type: "sliding_window"  # or "summary" for long conversations
    window_size: 5  # Number of turns to keep

  # Example reformulation
  # Turn 1: "What is RAG?" → search as-is
  # Turn 2: "How does it handle updates?" → reformulate: "How does RAG handle document updates?"

Phase 8: Evaluation Framework

RAG Evaluation is Non-Negotiable

If you're not measuring, you're guessing. Every production RAG system needs automated evaluation.

Evaluation Dimensions

Evaluation Dataset

Build a golden test set (minimum 100 examples):

eval_example:
  query: "What is the refund policy for enterprise customers?"
  expected_sources: ["policy-doc-v3.pdf", "enterprise-agreement.md"]
  expected_answer_contains:
    - "30-day refund window"
    - "written notice required"
    - "prorated for annual plans"
  answer_type: "factual"
  difficulty: "easy"  # easy / medium / hard

Test set composition:

• 40% easy (single document, direct answer)
• 35% medium (multiple documents, synthesis needed)
• 15% hard (requires reasoning, edge cases)
• 10% unanswerable (answer NOT in corpus — must detect)

LLM-as-Judge Prompts

Faithfulness (hallucination detection):

Given the context and the answer, determine if the answer is faithful to the context.

Context: {context}
Question: {question}
Answer: {answer}

Score 1-5:
1 = Contains fabricated information not in context
2 = Mostly faithful but includes unsupported claims
3 = Faithful with minor extrapolation
4 = Faithful, well-supported
5 = Perfectly faithful, every claim traceable to context

Score: [1-5]
Reasoning: [explain]

Answer Relevance:

Does this answer address the user's question?

Question: {question}
Answer: {answer}

Score 1-5:
1 = Completely irrelevant
2 = Partially relevant, misses key aspects
3 = Relevant but incomplete
4 = Relevant and mostly complete
5 = Perfectly addresses all aspects of the question

Score: [1-5]
Reasoning: [explain]

Evaluation Tools

Evaluation Cadence

Phase 9: Production Deployment

Production Architecture

┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│   Client     │────▶│  API Gateway  │────▶│  RAG Service  │
│   (App/API)  │     │  (Rate limit) │     │              │
└─────────────┘     └──────────────┘     │  Query Proc.  │
                                          │  Retrieval    │
                                          │  Reranking    │
                                          │  Generation   │
                                          └───────┬───────┘
                                                  │
                    ┌──────────────┐     ┌────────▼───────┐
                    │  Ingestion    │────▶│  Vector Store   │
                    │  Pipeline     │     │  + Metadata     │
                    └──────────────┘     └────────────────┘

Production Checklist

Pre-Launch (Mandatory):

• Golden test set passing (>85% on all dimensions)
• Hallucination rate < 5% on test set
• Latency P95 < target (typically 3-5s)
• Rate limiting configured
• Input validation (max query length, content filtering)
• Output filtering (PII detection, content safety)
• Error handling (vector DB down, LLM timeout, empty results)
• Fallback behavior defined ("I don't know" > hallucination)
• Logging and tracing enabled
• Cost monitoring and alerts set
• Load tested at 2x expected peak

Security:

• No prompt injection vectors (user input sanitized)
• Access control on documents (user sees only authorized content)
• No PII leakage across user boundaries
• API authentication required
• Rate limiting per user/API key
• Audit logging for compliance

Caching Strategy

caching:
  query_cache:
    type: "semantic"  # Cache semantically similar queries
    ttl: 3600  # 1 hour
    similarity_threshold: 0.95
    expected_hit_rate: "20-40%"

  embedding_cache:
    type: "exact"  # Cache document embeddings
    ttl: 86400  # 24 hours (or until document changes)

  llm_response_cache:
    type: "exact_query_context"
    ttl: 1800  # 30 minutes
    invalidate_on: "source_document_update"

Scaling Considerations

Phase 10: Monitoring & Observability

Production Monitoring Dashboard

rag_dashboard:
  real_time:
    - query_volume: "[queries/min]"
    - latency_p50: "[ms]"
    - latency_p95: "[ms]"
    - latency_p99: "[ms]"
    - error_rate: "[%]"
    - cache_hit_rate: "[%]"

  quality_signals:
    - retrieval_confidence_avg: "[0-1 — average similarity score]"
    - empty_retrieval_rate: "[% queries with no results above threshold]"
    - fallback_rate: "[% queries where model says 'I don't know']"
    - user_feedback_positive: "[% thumbs up]"
    - citation_rate: "[% answers with citations]"

  cost:
    - embedding_cost_daily: "[$]"
    - llm_cost_daily: "[$]"
    - reranker_cost_daily: "[$]"
    - vector_db_cost_daily: "[$]"
    - total_cost_per_query: "[$]"

  data_health:
    - index_freshness: "[time since last update]"
    - total_chunks_indexed: "[count]"
    - failed_ingestion_count: "[count]"
    - avg_chunk_quality_score: "[0-10]"

Alert Rules

Continuous Improvement Loop

Monitor → Identify Failure Patterns → Root Cause → Fix → Evaluate → Deploy

Weekly review questions:

1. What are the top 5 query types with lowest satisfaction?
2. Which documents are never retrieved? (potential indexing issues)
3. Which queries trigger "I don't know"? (coverage gaps)
4. What's the hallucination trend? (improving or degrading?)
5. Are costs trending up or down per query?

Phase 11: Advanced Patterns

Agentic RAG

User Query → Query Planner (LLM) → [Plan: search A, then search B, compare]
                                     ↓
                               Tool Execution
                               ├─ search_documents(query_A)
                               ├─ search_documents(query_B)
                               ├─ calculate(comparison)
                               └─ synthesize(results)
                                     ↓
                               Final Answer

When to use: Multi-step reasoning, cross-document comparison, calculation needed.

Implementation:

• Define tools: search_docs, lookup_entity, calculate, compare
• Use function calling with planning prompt
• Limit iterations (max 5 tool calls per query)
• Track and log the full reasoning chain

Graph RAG

graph_rag:
  when_to_use:
    - "Entity-heavy domains (legal, medical, organizational)"
    - "Queries about relationships ('who reports to X?')"
    - "Multi-hop reasoning ('what products use components from supplier Y?')"

  architecture:
    entities: "[Extract entities from documents]"
    relationships: "[Extract entity-entity relationships]"
    communities: "[Cluster entities into topic communities]"
    summaries: "[Generate community summaries]"

  retrieval:
    local_search: "Entity-focused — find specific entities and their neighbors"
    global_search: "Community-focused — synthesize across topic clusters"
    hybrid: "Combine vector similarity + graph traversal"

Corrective RAG (CRAG)

Query → Retrieve → Evaluate Relevance → 
  ├─ CORRECT: Retrieved docs are relevant → Generate answer
  ├─ AMBIGUOUS: Partially relevant → Refine query + re-retrieve
  └─ INCORRECT: Not relevant → Fall back to web search or "I don't know"

Self-RAG

Query → Retrieve → Generate + Self-Reflect →
  ├─ "Is retrieval needed?" → Skip if query is simple
  ├─ "Are results relevant?" → Re-retrieve if not
  ├─ "Is my answer supported?" → Revise if not faithful
  └─ "Is my answer useful?" → Regenerate if not

RAG + Fine-Tuning

Multi-Modal RAG

multimodal_rag:
  document_types:
    images: "Generate text descriptions via vision model; embed descriptions"
    tables: "Convert to structured text; embed as markdown"
    charts: "Describe in natural language; embed description"
    diagrams: "Generate detailed caption; store image reference + caption"

  retrieval:
    strategy: "Text-first retrieval with multimodal context assembly"
    image_in_context: "Include as base64 or URL reference in prompt"

Phase 12: Common Failure Modes & Fixes

Diagnostic Decision Tree

RAG quality is poor
├─ Retrieved chunks are irrelevant
│   ├─ Check: Chunking strategy → Are chunks self-contained?
│   ├─ Check: Embedding model → Run domain benchmark test
│   ├─ Check: Query transformation → Enable multi-query or HyDE
│   └─ Fix: Add reranking if not present
│
├─ Retrieved chunks are relevant but answer is wrong
│   ├─ Check: System prompt → Is grounding instruction clear?
│   ├─ Check: Context window → Is relevant info getting truncated?
│   ├─ Check: Conflicting sources → Add conflict resolution instructions
│   └─ Fix: Upgrade generation model
│
├─ System says "I don't know" too often
│   ├─ Check: Similarity threshold → Too high? Lower from 0.5 to 0.3
│   ├─ Check: Corpus coverage → Missing documents?
│   ├─ Check: top_k → Too low? Increase from 5 to 10
│   └─ Fix: Add query expansion
│
├─ Hallucination / makes things up
│   ├─ Check: System prompt → Add explicit grounding instructions
│   ├─ Check: Temperature → Set to 0.0-0.3 for factual tasks
│   ├─ Check: Retrieved context → Is it misleading or ambiguous?
│   └─ Fix: Add faithfulness evaluation in post-processing
│
└─ Too slow
    ├─ Check: Embedding latency → Batch? Cache?
    ├─ Check: Vector search → Index type? Quantization?
    ├─ Check: Reranker → Faster model or reduce candidate set
    └─ Fix: Add caching layer (semantic query cache)

10 RAG Anti-Patterns

10 Common Mistakes

Quality Scoring Rubric

RAG System Health Score (0-100)

Weighted Score: ___ / 100

Edge Cases

Low-Volume / Small Corpus

• Skip vector DB — use in-memory search or full-context stuffing
• Focus on chunking quality over retrieval sophistication
• Simple keyword + semantic hybrid is sufficient

High-Security / Regulated

• On-prem vector DB + self-hosted embedding model
• Document-level ACL enforcement at retrieval time
• Audit logging every query + response
• Data residency compliance for vector storage
• Consider homomorphic encryption for embeddings

Multi-Language

• Use multilingual embedding model (BGE-M3, Cohere embed-v4)
• Consider per-language indexes for large corpora
• Query language detection → route to appropriate index
• Cross-lingual retrieval: query in English, retrieve in any language

Real-Time / Streaming

• Event-driven ingestion (Kafka/webhooks → chunk → embed → index)
• Incremental indexing (add/update/delete individual chunks)
• Version management (don't serve partially indexed documents)
• Consider time-weighted scoring (recent docs ranked higher)

Very Large Corpus (>10M documents)

• Tiered retrieval: coarse filter → fine retrieval → reranking
• Hierarchical indexing (cluster → sub-cluster → document → chunk)
• Async processing pipeline with queue management
• Consider pre-computed answers for top 1000 queries

Natural Language Commands

When the user says... you respond with:

Built by AfrexAI — AI agents that compound capital and code. Zero dependencies. Pure methodology. Works with any RAG stack.