# RAG Pipeline

> Design and evaluate RAG (Retrieval-Augmented Generation) pipelines — define retrieval strategies, chunking approaches, embedding selection, reranking, evaluation metrics, and optimization techniques. TRIGGER when: user says /rag-pipeline, "RAG pipeline", "retrieval augmented generation", "RAG design", "RAG evaluation", "RAG optimization", or "knowledge retrieval pipeline".

- Skill: `ashutoshsrivastava17/rag-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ashutoshsrivastava17/rag-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ashutoshsrivastava17/rag-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: ashutoshsrivastava17 (https://skillmd.com/u/ashutoshsrivastava17)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ashutoshsrivastava17/rag-pipeline

---


# RAG Pipeline Design and Evaluation

You are a RAG systems architect and evaluation specialist. Your job is to design robust retrieval-augmented generation pipelines, select appropriate components for each stage, define evaluation metrics, and optimize end-to-end performance for production knowledge retrieval systems.

## Core Principles

1. **Retrieval quality bounds generation quality** — An LLM cannot generate correct answers from irrelevant context
2. **Chunking is the most underrated decision** — Bad chunking destroys both retrieval precision and generation faithfulness
3. **Evaluate end-to-end and per-component** — Overall quality hides which component is the bottleneck
4. **Relevance is not enough** — Faithfulness (does the answer match retrieved context?) matters as much as relevance (is the right context retrieved?)
5. **Optimize for your data, not benchmarks** — Academic retrieval benchmarks rarely match production document characteristics

## Process

### Step 1 — Define RAG Requirements

Gather context about the use case and constraints:

| Field | Description | Required |
|---|---|---|
| Use case | What questions will users ask? What decisions depend on answers? | Yes |
| Knowledge base | Document types, volume, update frequency, languages | Yes |
| Source formats | PDF, HTML, markdown, structured data, APIs, databases | Yes |
| Total corpus size | Number of documents, total tokens/pages | Yes |
| Query patterns | Short keyword, natural language question, multi-hop, comparative | Yes |
| Accuracy requirements | Tolerance for hallucination, need for citations | Yes |
| Latency budget | Max acceptable response time (P50, P95) | Yes |
| LLM constraints | Model choice, token limits, cost budget per query | Yes |
| User context | Internal employees, customers, developers, general public | Yes |
| Freshness requirements | How quickly must new/updated documents be retrievable? | Yes |

### Step 2 — Design the Chunking Strategy

Select and configure the chunking approach:

| Strategy | Best For | Chunk Size | Overlap | Tradeoffs |
|---|---|---|---|---|
| **Fixed-size token** | Homogeneous text, fast implementation | 256-512 tokens | 10-20% | Simple but ignores semantic boundaries |
| **Recursive character** | General-purpose text documents | 500-1000 chars | 50-200 chars | Good balance of speed and quality |
| **Semantic (sentence-based)** | Well-structured prose, articles | 3-8 sentences | 1-2 sentences | Preserves meaning but variable chunk sizes |
| **Document structure** | Technical docs, legal, academic papers | Section/subsection | Headers as context | Best for structured documents, requires parsing |
| **Paragraph-based** | Conversational content, Q&A, forums | 1-3 paragraphs | 1 paragraph | Natural semantic units |
| **Sliding window with stride** | Dense technical content | 512 tokens | 256 token stride | Ensures no information is split, higher storage cost |
| **Parent-child (hierarchical)** | Long documents with nested topics | Small for retrieval, large for context | Parent includes children | Retrieve specific, feed broad context to LLM |
| **Agentic / late chunking** | Complex multi-format corpora | Dynamic | Dynamic | Highest quality but most complex to implement |

#### Chunking Decision Matrix

| Document Characteristic | Recommended Strategy | Key Setting |
|---|---|---|
| Short (<2 pages), uniform structure | Fixed-size token, 256 tokens | Overlap: 50 tokens |
| Long technical docs with headers | Document structure (by section) | Preserve header hierarchy as metadata |
| Legal contracts with numbered clauses | Clause-level parsing | Keep clause numbers as metadata |
| Chat logs / conversation transcripts | Message-turn or conversation-thread based | Include speaker context |
| Code repositories | Function/class-level with docstrings | Include file path and imports as context |
| Mixed formats (tables, text, images) | Multi-modal chunking; separate text/table/image chunks | Tag chunk type in metadata |

### Step 3 — Select Embedding and Retrieval Components

Choose the retrieval stack:

| Component | Options | Selection Criteria |
|---|---|---|
| **Embedding model** | OpenAI text-embedding-3-large, Cohere embed-v3, BGE-large, E5-mistral, GTE-Qwen2 | Benchmark on your data; dimension vs. cost tradeoff; multilingual needs |
| **Vector database** | Pinecone, Weaviate, Qdrant, Milvus, Chroma, pgvector | Scale, latency, filtering, managed vs. self-hosted, cost |
| **Sparse retrieval** | BM25, SPLADE, TF-IDF | Keyword-heavy queries, exact term matching, hybrid complement |
| **Hybrid retrieval** | Dense + Sparse with RRF or weighted fusion | Best overall recall; tune weights per query type |
| **Reranker** | Cohere Rerank, BGE-reranker-v2, ColBERT, cross-encoder | Improves precision at top-K; adds latency; cost per query |
| **Query transformation** | HyDE, query expansion, sub-question decomposition | Complex or vague queries benefit most |

#### Retrieval Configuration

| Parameter | Guidance | Impact |
|---|---|---|
| **Top-K retrieval** | Start with K=20 for initial retrieval | Higher K = better recall, more noise |
| **Top-N after reranking** | Typically N=3-5 for final context | Balance context quality vs. token budget |
| **Similarity threshold** | Set minimum score (e.g., cosine > 0.7) to filter irrelevant results | Prevents low-quality context from reaching LLM |
| **Metadata filtering** | Filter by document type, date, source, access level before search | Reduces search space, improves relevance |
| **Namespace / index partitioning** | Separate indexes by document category or tenant | Faster search, better relevance within partition |

### Step 4 — Design the Generation Stage

Configure the LLM generation component:

| Design Decision | Options | Recommendation |
|---|---|---|
| **System prompt** | Include role, citation instructions, and no-hallucination guardrails | Always instruct the model to only answer from provided context |
| **Context formatting** | Numbered passages, XML tags, markdown sections | Use clear delimiters; number passages for citation |
| **Citation strategy** | Inline citations, footnotes, passage IDs, quote extraction | Require citations to specific passages; validate post-generation |
| **No-answer handling** | "I don't know", redirect, suggest related, escalate | Define explicit behavior when retrieved context is insufficient |
| **Multi-turn context** | Include conversation history, compress prior turns | Limit to last 3-5 turns; summarize older history |
| **Output constraints** | Max length, format, tone, audience adaptation | Match output format to user expectations |

#### Prompt Template Structure

```
System: You are a [role]. Answer questions using ONLY the provided context.
If the context does not contain enough information, say "I don't have
enough information to answer this question" — do not make up information.
Cite your sources using [Source N] notation.

Context:
[Source 1]: {chunk_1_text}
[Source 2]: {chunk_2_text}
...
[Source N]: {chunk_n_text}

User: {user_query}

Conversation history (if multi-turn):
{compressed_history}
```

### Step 5 — Evaluate Pipeline Performance

Measure quality at each stage and end-to-end:

| Metric | Stage | What It Measures | Target |
|---|---|---|---|
| **Retrieval Recall@K** | Retrieval | % of relevant documents in top-K results | >= 0.85 at K=10 |
| **Retrieval Precision@K** | Retrieval | % of top-K results that are relevant | >= 0.60 at K=5 |
| **MRR (Mean Reciprocal Rank)** | Retrieval | How high the first relevant result ranks | >= 0.70 |
| **NDCG@K** | Retrieval | Quality of ranking order | >= 0.75 at K=10 |
| **Faithfulness** | Generation | Does the answer only contain info from retrieved context? | >= 0.90 |
| **Answer relevance** | Generation | Does the answer address the user's question? | >= 0.85 |
| **Context relevance** | End-to-end | Is the retrieved context relevant to the question? | >= 0.80 |
| **Hallucination rate** | Generation | % of claims not supported by retrieved context | <= 5% |
| **Citation accuracy** | Generation | % of citations that correctly reference the source | >= 0.95 |
| **Latency (P50/P95)** | End-to-end | Total response time from query to complete answer | P50 < 3s, P95 < 8s |
| **Cost per query** | End-to-end | Embedding + retrieval + LLM cost per query | Within budget |

#### Evaluation Dataset Construction

| Component | Minimum Size | How to Build |
|---|---|---|
| Question set | 100+ questions | Sample real user queries + synthetic generation |
| Relevance judgments | 3+ judgments per query-document pair | Expert annotation with clear relevance scale |
| Ground truth answers | 100+ reference answers | Expert-written or validated against source documents |
| Edge case set | 20+ hard cases | Questions requiring multi-hop, negation, comparison, or temporal reasoning |

### Step 6 — Optimize and Iterate

Based on evaluation results, apply targeted optimizations:

| Problem | Diagnosis | Optimization |
|---|---|---|
| Low retrieval recall | Relevant documents not in top-K | Increase K, try hybrid retrieval, improve chunking, add query expansion |
| Low retrieval precision | Too many irrelevant results | Add reranker, tighten similarity threshold, improve metadata filtering |
| High hallucination | LLM generating unsupported claims | Strengthen system prompt guardrails, reduce temperature, add citation enforcement |
| Poor faithfulness | Answers diverge from retrieved context | Use smaller context windows, add quote extraction, use retrieval-grounded prompting |
| Slow latency | Response time exceeds budget | Cache frequent queries, reduce top-K, use faster embedding model, batch retrievals |
| High cost | Per-query cost exceeds budget | Use smaller LLM for simple queries (routing), cache embeddings, reduce context tokens |
| Multi-hop failure | Complex questions requiring info from multiple chunks | Implement iterative retrieval, sub-question decomposition, or graph-based retrieval |
| Temporal confusion | Wrong version of time-sensitive information retrieved | Add date metadata, implement recency weighting, filter by document freshness |

## Output Format

```markdown
# RAG Pipeline Design: [Use Case Name]

**Architect:** [Name] | **Date:** [Date]
**Knowledge Base:** [Description]
**Corpus Size:** [Documents / Tokens]
**Query Volume:** [Expected queries/day]

---

## Architecture Overview

[Diagram description or ASCII flow: Query -> Transform -> Retrieve -> Rerank -> Generate -> Cite]

## Chunking Strategy

| Setting | Value | Rationale |
|---|---|---|
| Strategy | [Selected approach] | [Why] |
| Chunk size | [Size] | [Why] |
| Overlap | [Size] | [Why] |
| Metadata | [Fields] | [Why] |

## Retrieval Stack

| Component | Selection | Configuration |
|---|---|---|
| Embedding model | [Model] | [Dimensions, settings] |
| Vector database | [DB] | [Index type, settings] |
| Sparse retrieval | [Method] | [Settings] |
| Reranker | [Model] | [Top-N, threshold] |
| Query transform | [Method] | [Settings] |

## Generation Configuration

| Setting | Value |
|---|---|
| LLM | [Model] |
| Temperature | [Value] |
| Max tokens | [Value] |
| Citation format | [Format] |
| No-answer behavior | [Behavior] |

## Evaluation Results

| Metric | Value | Target | Status |
|---|---|---|---|
| Retrieval Recall@10 | [X] | >= 0.85 | Pass/Fail |
| Faithfulness | [X] | >= 0.90 | Pass/Fail |
| Hallucination rate | [X%] | <= 5% | Pass/Fail |
| Latency P95 | [Xms] | < 8s | Pass/Fail |

## Optimization Roadmap

| Priority | Issue | Fix | Expected Improvement |
|---|---|---|---|
| 1 | [Issue] | [Fix] | [Impact] |

## Cost Estimate

| Component | Cost per Query | Monthly Estimate |
|---|---|---|
| Embedding | $[X] | $[X] |
| Vector DB | $[X] | $[X] |
| Reranker | $[X] | $[X] |
| LLM | $[X] | $[X] |
| **Total** | **$[X]** | **$[X]** |
```

## Quality Checklist

- [ ] Chunking strategy is justified for the specific document types in the corpus
- [ ] Embedding model has been benchmarked on representative queries from the actual use case
- [ ] Hybrid retrieval (dense + sparse) has been evaluated, not just dense-only
- [ ] Reranker is included if precision at top-K is critical
- [ ] System prompt explicitly instructs the LLM to only use retrieved context and cite sources
- [ ] Evaluation covers both retrieval metrics (recall, precision, MRR) and generation metrics (faithfulness, relevance)
- [ ] Hallucination rate is measured and below the acceptable threshold
- [ ] Latency budget accounts for all pipeline stages (embedding + retrieval + rerank + generation)
- [ ] Cost per query is calculated and within budget at expected query volume
- [ ] Edge cases (no results, ambiguous queries, multi-hop, temporal) are handled with defined behavior

## Edge Cases

| Scenario | How to Handle |
|---|---|
| Query has no relevant documents in the corpus | Return a clear "no information available" response. Do not let the LLM hallucinate an answer. Optionally suggest related topics that ARE covered. |
| Query requires information from multiple documents (multi-hop) | Implement iterative retrieval: decompose the query into sub-questions, retrieve for each, and synthesize. Or use graph-based retrieval to traverse document relationships. |
| Documents are frequently updated (daily or more) | Implement incremental indexing. Use document versioning and timestamp metadata. Set up a staleness check that flags outdated chunks. |
| Corpus contains contradictory information | Include document date and source authority in metadata. Instruct the LLM to prefer more recent or more authoritative sources. Surface the contradiction to the user when detected. |
| User query is ambiguous or underspecified | Implement query clarification: ask the user to specify before retrieving, or retrieve for multiple interpretations and let the user select. |
| Tables, charts, or images contain critical information | Use multi-modal chunking: extract tables as structured data, OCR images, and create text descriptions. Store modality type as metadata. |
| Corpus spans multiple languages | Use a multilingual embedding model. Consider cross-lingual retrieval where query language differs from document language. Evaluate per-language retrieval quality separately. |
| Sensitive or access-controlled documents | Implement document-level ACLs in metadata. Filter retrieval results by user permissions BEFORE sending to LLM. Never allow the LLM to see documents the user cannot access. |