RAG

RAG - Retrieval Augmented Generation

Build intelligent Q&A systems that ground LLM responses in your documents, reducing hallucinations and enabling knowledge updates without retraining.

When to Activate

This skill should be activated when the user:

• Wants to build a Q&A system over their documents
• Needs to reduce LLM hallucinations with factual grounding
• Asks about "chat with docs", "ChatGPT for my data", or similar
• Wants to keep knowledge up-to-date without fine-tuning
• Needs source attribution for answers

Interactive Flow

Before implementing, gather requirements through focused questions.

Step 1: Understand the Use Case

Ask ONE question at a time:

"What type of documents will users query?"

A) Internal knowledge base (policies, procedures, docs)
B) Customer-facing FAQ / support articles
C) Technical documentation (API docs, code)
D) Mixed / other

Which one? (A/B/C/D)

Based on answer, follow up:

Step 2: Clarify Constraints

"What's your priority?"

A) Accuracy first (willing to accept slower responses)
B) Speed first (good-enough answers, fast)
C) Cost first (minimize API calls)

Choose A, B, or C.

Step 3: Confirm Before Implementation

"Based on your requirements, I'll build:

- **Document type**: [from Step 1]
- **Chunk size**: [512 for general, 256 for FAQ, 1024 for technical]
- **Embedding**: text-embedding-3-small
- **Retrieval**: Top-5 with AUTOINDEX
- **LLM**: gpt-4o-mini

Does this look right? (yes / adjust [what])"

Decision Points During Implementation

Core Concepts

The RAG Paradigm

RAG decouples knowledge storage from reasoning capability:

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Documents  │───▶│  Retrieval  │───▶│    LLM      │
│  (Facts)    │    │  (Relevance)│    │  (Reasoning)│
└─────────────┘    └─────────────┘    └─────────────┘
      ▲                   │                  │
      │                   ▼                  ▼
   Update            Top-K chunks        Answer with
   anytime           as context          citations

Key insight: LLMs are excellent reasoners but unreliable knowledge stores. RAG leverages their reasoning while externalizing knowledge to a retrievable corpus.

Mental Model: Library + Librarian

Think of RAG as a library (vector database) with a librarian (retrieval) helping a scholar (LLM):

• The library stores books (document chunks) indexed by topic (embeddings)
• The librarian finds relevant books based on the question
• The scholar synthesizes an answer from the provided materials

Why RAG Over Alternatives

Choose RAG when:

• Knowledge changes frequently (docs updated weekly/monthly)
• Users need source attribution ("where did you get this?")
• Corpus exceeds context window (>100K tokens)
• Domain accuracy matters more than response speed

Avoid RAG when:

• Corpus is tiny (<10 pages) — just use long context
• Questions don't need specific facts — pure LLM suffices
• Latency is critical (<100ms) — consider caching or fine-tuning

Pipeline Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        INDEXING PHASE                            │
├──────────────────────────────────────────────────────────────────┤
│  Documents  ──▶  Chunking  ──▶  Embedding  ──▶  Vector Store    │
│   (raw)         (512 tokens)   (1536-dim)      (Milvus)         │
└──────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────┐
│                        QUERY PHASE                               │
├──────────────────────────────────────────────────────────────────┤
│  Question  ──▶  Embed  ──▶  Search  ──▶  Top-K  ──▶  LLM  ──▶  Answer
│                           (HNSW)       chunks      (GPT-4)      │
└──────────────────────────────────────────────────────────────────┘

Stage Breakdown

Implementation

Core implementation with production-ready defaults:

from pymilvus import MilvusClient, DataType
from langchain.text_splitter import RecursiveCharacterTextSplitter
from openai import OpenAI

class RAGSystem:
    def __init__(self, collection_name: str = "rag_kb", uri: str = "./milvus.db"):
        self.client = MilvusClient(uri=uri)
        self.collection_name = collection_name
        self.openai = OpenAI()
        self.splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
        self._init_collection()

    def _embed(self, texts: list) -> list:
        response = self.openai.embeddings.create(model="text-embedding-3-small", input=texts)
        return [item.embedding for item in response.data]

    def _init_collection(self):
        if self.client.has_collection(self.collection_name):
            return
        schema = self.client.create_schema(auto_id=True, enable_dynamic_field=True)
        schema.add_field("id", DataType.INT64, is_primary=True)
        schema.add_field("text", DataType.VARCHAR, max_length=65535)
        schema.add_field("source", DataType.VARCHAR, max_length=512)
        schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=1536)

        index_params = self.client.prepare_index_params()
        index_params.add_index("embedding", index_type="AUTOINDEX", metric_type="COSINE")
        self.client.create_collection(self.collection_name, schema=schema, index_params=index_params)

    def add_document(self, text: str, source: str = ""):
        chunks = self.splitter.split_text(text)
        embeddings = self._embed(chunks)
        data = [{"text": c, "source": source, "embedding": e} for c, e in zip(chunks, embeddings)]
        self.client.insert(self.collection_name, data)
        return len(chunks)

    def query(self, question: str, top_k: int = 5):
        # Retrieve
        results = self.client.search(self.collection_name, self._embed([question]),
                                     limit=top_k, output_fields=["text", "source"])
        contexts = [{"text": h["entity"]["text"], "source": h["entity"]["source"]} for h in results[0]]

        # Generate
        context_text = "\n\n".join([f"[{c['source']}]: {c['text']}" for c in contexts])
        response = self.openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Answer based on these references. Cite sources.

References:
{context_text}

Question: {question}"""}],
            temperature=0.3
        )
        return {"answer": response.choices[0].message.content,
                "sources": list(set(c["source"] for c in contexts))}

Usage:

rag = RAGSystem()
rag.add_document(open("docs/intro.md").read(), source="intro.md")
result = rag.query("What is Milvus?")
print(result["answer"])

For advanced patterns (streaming, multi-turn, hybrid search), see references/advanced-patterns.md.

Configuration Guide

Chunk Size Selection

Rule of thumb: Start with 512/50, adjust based on answer quality.

Top-K Selection

Embedding Model Tradeoffs

See references/embedding-models.md for detailed comparison.

Common Pitfalls

1. Chunks Too Large

Symptom: Irrelevant information pollutes context Fix: Reduce chunk_size, or use semantic chunking

2. Chunks Too Small

Symptom: Answers lack context, feel fragmented Fix: Increase chunk_size or overlap

3. Wrong Embedding Model for Language

Symptom: Poor retrieval for non-English text Fix: Use multilingual model (bge-m3) or language-specific model

4. Ignoring Metadata

Symptom: Can't filter by date, source, or category Fix: Store metadata fields, use filtered search

5. No Source Attribution

Symptom: Users don't trust answers Fix: Always return sources, include in prompt

When to Level Up

References

Internal:

• references/chunk-strategies.md - Detailed chunking approaches
• references/embedding-models.md - Model comparison and selection
• references/advanced-patterns.md - Streaming, multi-turn, hybrid search

Core operators:

• core:chunking - Document chunking utilities
• core:embedding - Embedding generation
• core:ray - Data processing at scale

Verticals:

• verticals/enterprise-kb.md - Enterprise knowledge base
• verticals/customer-service.md - Customer service bot