Chunking Strategy Guide

Chunking Strategy Guide — RAG Document Chunking Strategy

A skill that enhances the data preprocessing capabilities of the rag-architect.

Target Agents

• rag-architect — Effectively chunks documents to improve retrieval quality
• eval-specialist — Evaluates the retrieval quality of chunking strategies

Chunking Strategy Comparison

Chunking Parameter Guide

Optimal Chunk Sizes

| Document Type | Chunk Size | Overlap | Rationale |
|--------------|-----------|---------|-----------|
| FAQ | 100-200 tokens | 0 | Q&A pairs are short |
| Technical docs | 300-500 tokens | 50 | Code + explanation units |
| Legal documents | 500-800 tokens | 100 | Article units |
| Academic papers | 400-600 tokens | 80 | Paragraph units |
| Chat logs | 200-300 tokens | 30 | Conversation turn units |
| Fiction/essays | 300-500 tokens | 50 | Scene/paragraph units |

Overlap Ratio Formula

optimal_overlap = chunk_size * 0.1 to 0.2

Rules:
- Independent documents (FAQ): Overlap 0
- Sequential documents (manuals): 10-15%
- Dense documents (legal): 15-20%
- Maximum overlap: Never exceed 25% of chunk_size

Semantic Chunking Algorithm

def semantic_chunking(text, model, threshold=0.5):
    """
    1. Split into sentences
    2. Calculate cosine similarity of adjacent sentence embeddings
    3. Split at points where similarity falls below threshold
    4. Apply min/max chunk size constraints
    """
    sentences = split_sentences(text)
    embeddings = model.encode(sentences)

    breakpoints = []
    for i in range(len(embeddings) - 1):
        sim = cosine_similarity(embeddings[i], embeddings[i+1])
        if sim < threshold:
            breakpoints.append(i + 1)

    chunks = split_at(sentences, breakpoints)
    return enforce_size_limits(chunks, min=100, max=800)

Per-Document-Type Preprocessing Pipeline

PDF

PDF > Text extraction (pdfplumber/pymupdf)
> Remove headers/footers
> Remove page numbers
> Tables > Markdown conversion
> Images > alt text / OCR
> Metadata extraction (title, author, date)
> Chunking

HTML/Webpages

HTML > Body extraction (trafilatura/readability)
> Remove navigation/sidebar/ads
> Markdown conversion
> Preserve link text ([text](URL))
> Preserve tables
> Metadata extraction (title, description)
> Chunking

Code

Code > AST parsing
> Split by function/class
> Docstring + signature + body
> Add file path metadata
> Link related test code
> Chunking (function-level)

Metadata Enrichment Strategy

chunk_with_metadata = {
    "text": "Chunk text...",
    "metadata": {
        "source": "document.pdf",
        "page": 5,
        "section": "3.2 Architecture",
        "heading_hierarchy": ["3. Design", "3.2 Architecture"],
        "chunk_index": 12,
        "total_chunks": 45,
        "created_at": "2025-01-15",
        "document_type": "technical_spec",
        "language": "en"
    }
}

Chunking Quality Evaluation Metrics

Embedding Model Selection Guide