Token Aware Chunking

Token-Aware Semantic Chunking Protocol (Boundary-Aligned RAG Slicing)

Overview

When indexing documentation or codebases for vector search and RAG retrieval, naive splitters slice text by fixed character counts (text[i:i+2000]).

Fixed-character chunking causes severe retrieval degradations:

1. Broken Code Blocks: Splits a TypeScript interface or Python function midway through its body, creating unparseable syntax fragments.
2. Mid-Word Token Clipping: Slices words across token boundaries, corrupting embedding vector representations.
3. Embedding Model Ceiling Exceedance: A character count that translates to 8,250 tokens gets silently truncated by an embedding model with an 8,192-token ceiling.

The Token-Aware Semantic Chunking Protocol measures chunk size strictly using the target tokenizer (tiktoken / BPE) and splits text recursively along semantic boundaries (Markdown Headers, AST Function Blocks, Double Newlines).

Fixed-Character Slicing vs. Token-Aware Semantic Chunking

┌─────────────────────────────────────────────────────────────┐
│                 Text Chunking Mechanics                     │
│                                                             │
│  Fixed Character Slicing (`len(text) == 2000`):             │
│  • Chunk 1 ends: `function calculateTotal(price: num`       │
│  • Chunk 2 starts: `ber, tax: number) { return price + ...` │
│  ↳ Syntax broken across 2 chunks! Vector embedding corrupted│
│                                                             │
│  Token-Aware Semantic Slicing (512 Tokens / AST Boundary):  │
│  • Chunk 1: Complete `calculateTotal` function + docstring  │
│  • Chunk 2: Complete `processPayment` function              │
│  ↳ 100% Valid code syntax, exact 512-token budget adherence │
└─────────────────────────────────────────────────────────────┘

The 4-Tier Semantic Split Hierarchy

When partitioning text into token-bounded chunks, search for split delimiters in descending priority:

┌───────────────────────────────────────────────────────────────────────────┐
│ 1. SECTION BOUNDARIES: Markdown Headers (`\n## `, `\n### `), Class defs   │
│ 2. BLOCK BOUNDARIES: Double Newlines (`\n\n`), Function definitions       │
│ 3. STATEMENT BOUNDARIES: Single Newlines (`\n`), Semicolons               │
│ 4. FALLBACK: Word whitespace (` `) (Never split in the middle of a token) │
└───────────────────────────────────────────────────────────────────────────┘

Production Python Token-Aware Semantic Chunker

import tiktoken
from typing import List

def chunk_text_token_aware(
    text: str,
    max_tokens: int = 512,
    overlap_tokens: int = 50,
    model_name: str = "gpt-4o"
) -> List[str]:
    """Recursively splits markdown/code along semantic boundaries to fit token limits."""
    enc = tiktoken.encoding_for_model(model_name)
    tokens = enc.encode(text)
    
    if len(tokens) <= max_tokens:
        return [text]

    chunks = []
    # Split recursively by semantic delimiters
    delimiters = ["\n## ", "\n### ", "\n\n", "\n", " "]
    
    def recursive_split(sub_text: str) -> List[str]:
        sub_tokens = enc.encode(sub_text)
        if len(sub_tokens) <= max_tokens:
            return [sub_text]
            
        for delim in delimiters:
            if delim in sub_text:
                parts = sub_text.split(delim)
                accumulated = ""
                sub_chunks = []
                
                for part in parts:
                    candidate = f"{accumulated}{delim}{part}" if accumulated else part
                    if len(enc.encode(candidate)) <= max_tokens:
                        accumulated = candidate
                    else:
                        if accumulated:
                            sub_chunks.append(accumulated)
                        accumulated = part
                if accumulated:
                    sub_chunks.append(accumulated)
                    
                return sub_chunks
                
        # Absolute fallback: Token slice
        raw_toks = enc.encode(sub_text)
        return [enc.decode(raw_toks[i:i+max_tokens]) for i in range(0, len(raw_toks), max_tokens - overlap_tokens)]

    return recursive_split(text)

Benchmark Comparison

Indexing 500 pages of technical documentation for vector search:

Agent Operational Directive

MANDATORY: Knowledge ingestion pipelines and RAG indexers must measure chunk sizes using target tokenizer encoders (tiktoken). Never split text by raw character counts; always split along Markdown header and AST block boundaries.