RAG Architect
Core Workflow
- Requirements Analysis — Identify retrieval needs, latency constraints, accuracy requirements, and scale
- Vector Store Design — Select database, schema design, indexing strategy, sharding approach
- Chunking Strategy — Document splitting, overlap, semantic boundaries, metadata enrichment
- Retrieval Pipeline — Embedding selection, query transformation, hybrid search, reranking
- Evaluation & Iteration — Metrics tracking, retrieval debugging, continuous optimization
For each step, validate before moving on (see checkpoints below).
Reference Guide
Load detailed guidance based on context:
| Topic |
Reference |
Load When |
| Vector Databases |
references/vector-databases.md |
Comparing Pinecone, Weaviate, Chroma, pgvector, Qdrant |
| Embedding Models |
references/embedding-models.md |
Selecting embeddings, fine-tuning, dimension trade-offs |
| Chunking Strategies |
references/chunking-strategies.md |
Document splitting, overlap, semantic chunking |
| Retrieval Optimization |
references/retrieval-optimization.md |
Hybrid search, reranking, query expansion, filtering |
| RAG Evaluation |
references/rag-evaluation.md |
Metrics, evaluation frameworks, debugging retrieval |
Implementation Examples
1. Chunking Documents
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Evaluate chunk_size on your domain data — never use 512 blindly
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=100,
separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.create_documents(
texts=[doc.page_content for doc in raw_docs],
metadatas=[{"source": doc.metadata["source"], "timestamp": doc.metadata.get("timestamp")} for doc in raw_docs],
)
Checkpoint: assert all(c.metadata.get("source") for c in chunks), "Missing source metadata"
2. Generating Embeddings & Indexing
from openai import OpenAI
import qdrant_client
from qdrant_client.models import VectorParams, Distance, PointStruct
client = OpenAI()
qdrant = qdrant_client.QdrantClient("localhost", port=6333)
# Create collection
qdrant.recreate_collection(
collection_name="knowledge_base",
vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)
def embed_chunks(chunks: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
response = client.embeddings.create(input=chunks, model=model)
return [r.embedding for r in response.data]
# Idempotent upsert with deduplication via deterministic IDs
import hashlib, uuid
points = []
for i, chunk in enumerate(chunks):
doc_id = str(uuid.UUID(hashlib.md5(chunk.page_content.encode()).hexdigest()))
embedding = embed_chunks([chunk.page_content])[0]
points.append(PointStruct(id=doc_id, vector=embedding, payload=chunk.metadata))
qdrant.upsert(collection_name="knowledge_base", points=points)
Checkpoint: assert qdrant.count("knowledge_base").count == len(set(p.id for p in points)), "Deduplication failed"
3. Hybrid Search (Vector + BM25)
from qdrant_client.models import Filter, FieldCondition, MatchValue, SparseVector
from rank_bm25 import BM25Okapi
def hybrid_search(query: str, tenant_id: str, top_k: int = 20) -> list:
# Dense retrieval
query_embedding = embed_chunks([query])[0]
tenant_filter = Filter(must=[FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id))])
dense_results = qdrant.search(
collection_name="knowledge_base",
query_vector=query_embedding,
query_filter=tenant_filter,
limit=top_k,
)
# Sparse retrieval (BM25)
corpus = [r.payload.get("text", "") for r in dense_results]
bm25 = BM25Okapi([doc.split() for doc in corpus])
bm25_scores = bm25.get_scores(query.split())
# Reciprocal Rank Fusion
ranked = sorted(
zip(dense_results, bm25_scores),
key=lambda x: 0.6 * x[0].score + 0.4 * x[1],
reverse=True,
)
return [r for r, _ in ranked[:top_k]]
Checkpoint: assert len(hybrid_search("test query", tenant_id="demo")) > 0, "Hybrid search returned no results"
4. Reranking Top-K Results
Load provider API keys from environment variables or a secrets manager; never commit them to source code.
import os
import cohere
co = cohere.Client(os.environ["COHERE_API_KEY"])
def rerank(query: str, results: list, top_n: int = 5) -> list:
docs = [r.payload.get("text", "") for r in results]
reranked = co.rerank(query=query, documents=docs, top_n=top_n, model="rerank-english-v3.0")
return [results[r.index] for r in reranked.results]
5. Retrieval Evaluation
# Run precision@k and recall@k against a labeled evaluation set
# python evaluate.py --metrics precision@10 recall@10 mrr --collection knowledge_base
from ragas import evaluate
from ragas.metrics import context_precision, context_recall, faithfulness, answer_relevancy
from datasets import Dataset
eval_dataset = Dataset.from_dict({
"question": questions,
"contexts": retrieved_contexts,
"answer": generated_answers,
"ground_truth": ground_truth_answers,
})
results = evaluate(eval_dataset, metrics=[context_precision, context_recall, faithfulness, answer_relevancy])
print(results)
Checkpoint: Target context_precision >= 0.7 and context_recall >= 0.6 before moving to LLM integration.
Constraints
MUST DO
- Evaluate multiple embedding models on your domain data before committing
- Implement hybrid search (vector + keyword) for production systems
- Add metadata filters for multi-tenant or domain-specific retrieval
- Measure retrieval metrics (precision@k, recall@k, MRR, NDCG)
- Use reranking for top-k results before passing context to LLM
- Implement idempotent ingestion with deduplication (deterministic IDs)
- Monitor retrieval latency and quality over time
- Version embeddings and plan for model migration
MUST NOT DO
- Use default chunk size (512) without evaluation on your domain data
- Skip metadata enrichment (source, timestamp, section)
- Ignore retrieval quality metrics in favor of only LLM output quality
- Store raw documents without preprocessing/cleaning
- Use cosine similarity alone for complex multi-domain retrieval
- Deploy without testing on production-like data volumes
- Forget to handle edge cases (empty results, malformed docs)
- Couple the embedding model tightly to application code
Output Templates
When designing RAG architecture, deliver:
- System architecture diagram (ingestion + retrieval pipelines)
- Vector database selection with trade-off analysis
- Chunking strategy with examples and rationale
- Retrieval pipeline design (query → results flow)
- Evaluation plan with metrics, benchmarks, and pass/fail thresholds
Maintained by @jeffallan, Principal Consultant at Synergetic Solutions
Documentation
1---2name: rag-architect3description: Designs and implements production-grade RAG systems by chunking documents, generating embeddings, configuring vector stores, building hybrid search pipelines, applying reranking, and evaluating retrieval quality. Use when building RAG systems, vector databases, or knowledge-grounded AI applications requiring semantic search, document retrieval, context augmentation, similarity search, or embedding-based indexing.4license: MIT5metadata:6 source: https://skillmd.com/skills/tuyv/rag-architect7 author: https://github.com/Jeffallan8 company: https://synergetic.solutions9 version: "1.1.0"10 domain: data-ml11 triggers: RAG, retrieval-augmented generation, vector search, embeddings, semantic search, vector database, document retrieval, knowledge base, context retrieval, similarity search12 role: architect13 scope: system-design14 output-format: architecture15 related-skills: python-pro, database-optimizer, monitoring-expert, api-designer16---1718# RAG Architect1920## Core Workflow21221. **Requirements Analysis** — Identify retrieval needs, latency constraints, accuracy requirements, and scale232. **Vector Store Design** — Select database, schema design, indexing strategy, sharding approach243. **Chunking Strategy** — Document splitting, overlap, semantic boundaries, metadata enrichment254. **Retrieval Pipeline** — Embedding selection, query transformation, hybrid search, reranking265. **Evaluation & Iteration** — Metrics tracking, retrieval debugging, continuous optimization2728For each step, validate before moving on (see checkpoints below).2930## Reference Guide3132Load detailed guidance based on context:3334| Topic | Reference | Load When |35|-------|-----------|-----------|36| Vector Databases | `references/vector-databases.md` | Comparing Pinecone, Weaviate, Chroma, pgvector, Qdrant |37| Embedding Models | `references/embedding-models.md` | Selecting embeddings, fine-tuning, dimension trade-offs |38| Chunking Strategies | `references/chunking-strategies.md` | Document splitting, overlap, semantic chunking |39| Retrieval Optimization | `references/retrieval-optimization.md` | Hybrid search, reranking, query expansion, filtering |40| RAG Evaluation | `references/rag-evaluation.md` | Metrics, evaluation frameworks, debugging retrieval |4142## Implementation Examples4344### 1. Chunking Documents4546```python47from langchain.text_splitter import RecursiveCharacterTextSplitter4849# Evaluate chunk_size on your domain data — never use 512 blindly50splitter = RecursiveCharacterTextSplitter(51 chunk_size=800,52 chunk_overlap=100,53 separators=["\n\n", "\n", ". ", " "],54)5556chunks = splitter.create_documents(57 texts=[doc.page_content for doc in raw_docs],58 metadatas=[{"source": doc.metadata["source"], "timestamp": doc.metadata.get("timestamp")} for doc in raw_docs],59)60```6162**Checkpoint:** `assert all(c.metadata.get("source") for c in chunks), "Missing source metadata"`6364### 2. Generating Embeddings & Indexing6566```python67from openai import OpenAI68import qdrant_client69from qdrant_client.models import VectorParams, Distance, PointStruct7071client = OpenAI()72qdrant = qdrant_client.QdrantClient("localhost", port=6333)7374# Create collection75qdrant.recreate_collection(76 collection_name="knowledge_base",77 vectors_config=VectorParams(size=1536, distance=Distance.COSINE),78)7980def embed_chunks(chunks: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:81 response = client.embeddings.create(input=chunks, model=model)82 return [r.embedding for r in response.data]8384# Idempotent upsert with deduplication via deterministic IDs85import hashlib, uuid8687points = []88for i, chunk in enumerate(chunks):89 doc_id = str(uuid.UUID(hashlib.md5(chunk.page_content.encode()).hexdigest()))90 embedding = embed_chunks([chunk.page_content])[0]91 points.append(PointStruct(id=doc_id, vector=embedding, payload=chunk.metadata))9293qdrant.upsert(collection_name="knowledge_base", points=points)94```9596**Checkpoint:** `assert qdrant.count("knowledge_base").count == len(set(p.id for p in points)), "Deduplication failed"`9798### 3. Hybrid Search (Vector + BM25)99100```python101from qdrant_client.models import Filter, FieldCondition, MatchValue, SparseVector102from rank_bm25 import BM25Okapi103104def hybrid_search(query: str, tenant_id: str, top_k: int = 20) -> list:105 # Dense retrieval106 query_embedding = embed_chunks([query])[0]107 tenant_filter = Filter(must=[FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id))])108 dense_results = qdrant.search(109 collection_name="knowledge_base",110 query_vector=query_embedding,111 query_filter=tenant_filter,112 limit=top_k,113 )114115 # Sparse retrieval (BM25)116 corpus = [r.payload.get("text", "") for r in dense_results]117 bm25 = BM25Okapi([doc.split() for doc in corpus])118 bm25_scores = bm25.get_scores(query.split())119120 # Reciprocal Rank Fusion121 ranked = sorted(122 zip(dense_results, bm25_scores),123 key=lambda x: 0.6 * x[0].score + 0.4 * x[1],124 reverse=True,125 )126 return [r for r, _ in ranked[:top_k]]127```128129**Checkpoint:** `assert len(hybrid_search("test query", tenant_id="demo")) > 0, "Hybrid search returned no results"`130131### 4. Reranking Top-K Results132133Load provider API keys from environment variables or a secrets manager; never commit them to source code.134135```python136import os137138import cohere139140co = cohere.Client(os.environ["COHERE_API_KEY"])141142def rerank(query: str, results: list, top_n: int = 5) -> list:143 docs = [r.payload.get("text", "") for r in results]144 reranked = co.rerank(query=query, documents=docs, top_n=top_n, model="rerank-english-v3.0")145 return [results[r.index] for r in reranked.results]146```147148### 5. Retrieval Evaluation149150```python151# Run precision@k and recall@k against a labeled evaluation set152# python evaluate.py --metrics precision@10 recall@10 mrr --collection knowledge_base153154from ragas import evaluate155from ragas.metrics import context_precision, context_recall, faithfulness, answer_relevancy156from datasets import Dataset157158eval_dataset = Dataset.from_dict({159 "question": questions,160 "contexts": retrieved_contexts,161 "answer": generated_answers,162 "ground_truth": ground_truth_answers,163})164165results = evaluate(eval_dataset, metrics=[context_precision, context_recall, faithfulness, answer_relevancy])166print(results)167```168169**Checkpoint:** Target `context_precision >= 0.7` and `context_recall >= 0.6` before moving to LLM integration.170171## Constraints172173### MUST DO174- Evaluate multiple embedding models on your domain data before committing175- Implement hybrid search (vector + keyword) for production systems176- Add metadata filters for multi-tenant or domain-specific retrieval177- Measure retrieval metrics (precision@k, recall@k, MRR, NDCG)178- Use reranking for top-k results before passing context to LLM179- Implement idempotent ingestion with deduplication (deterministic IDs)180- Monitor retrieval latency and quality over time181- Version embeddings and plan for model migration182183### MUST NOT DO184- Use default chunk size (512) without evaluation on your domain data185- Skip metadata enrichment (source, timestamp, section)186- Ignore retrieval quality metrics in favor of only LLM output quality187- Store raw documents without preprocessing/cleaning188- Use cosine similarity alone for complex multi-domain retrieval189- Deploy without testing on production-like data volumes190- Forget to handle edge cases (empty results, malformed docs)191- Couple the embedding model tightly to application code192193## Output Templates194195When designing RAG architecture, deliver:1961. System architecture diagram (ingestion + retrieval pipelines)1972. Vector database selection with trade-off analysis1983. Chunking strategy with examples and rationale1994. Retrieval pipeline design (query → results flow)2005. Evaluation plan with metrics, benchmarks, and pass/fail thresholds201202Maintained by [@jeffallan](https://github.com/jeffallan), Principal Consultant at [Synergetic Solutions](https://synergetic.solutions)203204[Documentation](https://jeffallan.github.io/claude-skills/skills/data-ml/rag-architect/)