aselimc RAG Pipeline
Architecture
Documents -> Chunking -> Embedding -> Vector Store -> Retrieval -> LLM Generation
Chunking Strategies
- Fixed-size: 512 tokens with 50-token overlap
- Recursive: split by
\n\n>\n>.> - Semantic: split by topic/section boundaries
Embedding & Retrieval
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(chunks)
# Store in FAISS/Qdrant/Chroma
Hybrid Retrieval (Best Practice)
- BM25 keyword search (sparse)
- Dense vector search (semantic)
- Reciprocal rank fusion to merge results
- Cross-encoder re-ranking on top-k
Rules
- Chunk size should match embedding model's context window
- Always include source metadata for attribution
- Evaluate: retrieval recall@k, answer accuracy, faithfulness
- Guard against prompt injection in retrieved content
Key Libraries
langchain, llama-index, sentence-transformers, FAISS, chromadb