RAG And Vector Search

RAG and Vector Search

Design, implement, and evaluate retrieval-augmented generation pipelines.

Core principle: Retrieval quality determines answer quality. A great generator with poor retrieval produces confident hallucinations.

When to Use

• Building a question-answering system over documents
• Adding knowledge retrieval to an LLM application
• Implementing semantic search
• Evaluating or debugging an existing RAG pipeline
• Choosing between vector databases
• Optimizing retrieval relevance or reducing hallucination

When NOT to Use

• Fine-tuning a model (RAG and fine-tuning are complementary, not substitutes)
• Simple keyword search (use Elasticsearch/Algolia)
• Real-time data that can't be pre-indexed (use function calling instead)

Pipeline Architecture

INGESTION
  Documents → Clean → Chunk → Embed → Store in vector DB

RETRIEVAL  
  Query → Embed → Similarity Search → Rerank → Select Context

GENERATION
  Context + Query → LLM → Answer

Document Chunking

Strategy Selection

Chunking Rules

• Always include metadata: source, section title, page number, document date
• Never split mid-sentence
• Maintain 10-20% overlap between consecutive chunks
• Test with 3+ strategies and measure retrieval precision before choosing

def chunk_fixed(text: str, size=1024, overlap=100) -> list[str]:
    chunks, step = [], size - overlap
    for i in range(0, len(text), step):
        chunks.append(text[i:i+size])
    return chunks

Embedding Model Selection

Selection criteria:

1. Latency requirement < 100ms → use fast model (MiniLM)
2. High precision needed → use larger model (mpnet, ada-002)
3. Specialized content → use domain-specific model
4. Self-hosted required → use open-source (MiniLM, mpnet)

Vector Database Selection

Retrieval Strategies

Dense (Semantic) Retrieval

results = vector_db.query(
    query_embedding=embed(query),
    k=5,
    filter={"category": "orders"}  # metadata pre-filtering
)

Best for: semantic similarity, paraphrase matching, conceptual queries.

Sparse (Keyword) Retrieval

BM25 or Elasticsearch. Best for: exact terminology, proper nouns, product codes.

Hybrid Search (Recommended for Production)

Combine dense + sparse with Reciprocal Rank Fusion:

dense_results = semantic_search(query, k=10)
sparse_results = bm25_search(query, k=10)
fused = reciprocal_rank_fusion([dense_results, sparse_results])

Reranking (Two-Stage)

1. Retrieve k=20 candidates with fast dense search
2. Rerank with cross-encoder model (slower but more accurate)
3. Return top k=5

Use when: precision matters more than latency. Adds ~100-200ms.

Query Transformation

Context Window Optimization

Assembly Strategy

1. Rank chunks by relevance score (highest first)
2. Remove redundant chunks (>90% semantic overlap)
3. Fit within token budget (leave room for prompt + output)
4. Order: general context first, specific evidence last

Context Compression

When too many relevant chunks exist:

• Summarize lower-ranked chunks (keep key facts, remove prose)
• Extract only relevant sentences from each chunk
• Set minimum relevance threshold (drop chunks below 0.7 similarity)

RAG Evaluation

Key Metrics

RAGAS Framework

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

scores = evaluate(
    dataset=test_dataset,  # questions, contexts, answers, ground_truths
    metrics=[faithfulness, answer_relevancy, context_precision],
)
print(scores)  # faithfulness: 0.92, answer_relevancy: 0.88, ...

Common Failure Modes

Common Mistakes

• Not testing multiple chunking strategies before choosing
• Using generic embedding model for specialized domain without benchmarking
• Single-stage dense retrieval only (missing keyword matches)
• No evaluation framework (can't measure quality improvements)
• No fallback when retrieval returns 0 results
• Caching responses without checking if underlying documents changed

Verification Checklist

• Chunking strategy tested and retrieval precision >0.70
• Embedding model benchmarked for domain relevance
• Hybrid search implemented (dense + sparse)
• Reranking applied for high-precision use cases
• Faithfulness >0.90 on test set (no hallucination)
• Context relevance >0.80 on test set
• Token budget respected (context fits in window)
• Fallback for empty retrieval results
• Latency <500ms end-to-end (retrieval + generation)
• Evaluation metrics monitored in production