Intelligent Text Chunking

Intelligent Text Chunking

There is a real implementation at src/utils/intelligent_chunker.py. Use it; don't roll your own splitter.

API

from src.utils.intelligent_chunker import IntelligentTextChunker, ChunkType, TextChunk

chunker = IntelligentTextChunker(
    max_chunk_size=512,      # tokens/characters per chunk
    min_chunk_size=50,
    overlap_ratio=0.1,       # 0.0–0.5
    preserve_sentences=True,
    preserve_paragraphs=True,
)
chunks: list[TextChunk] = chunker.chunk(text)
# Each TextChunk has: content, start_position, end_position, chunk_id,
#   chunk_type (ChunkType enum), metadata, overlap_with_previous, overlap_with_next

ChunkType enum: SEMANTIC, STRUCTURAL, FIXED_SIZE, SLIDING_WINDOW (intelligent_chunker.py).

The chunker has built-in awareness of:

• Multi-language sentence boundaries (English, Chuukese, generic CJK punctuation).
• Structure markers (markdown headings, list items, dictionary entries, page breaks).
• Semantic transition phrases (topic change, continuation, conclusion, examples).

Where it's used

• EnhancedOCRProcessor — chunks OCR output for downstream training/storage.
• LargeDocumentProcessor — top-level pipeline for 200+ page documents.
• AITrainingDataGenerator consumes chunks indirectly through ParsedDocument.

Critical: scripture references aren't protected

The chunker does not know about Bible references — it will happily split 1 Cor. 13:4-7 across chunks. Always wrap calls with protect_scripture_references / restore_scripture_references from src/utils/scripture_parser.py when input may contain them. See the scripture-reference-parsing skill.

from src.utils.scripture_parser import (
    protect_scripture_references, restore_scripture_references
)

protected, refs = protect_scripture_references(raw)
chunks = chunker.chunk(protected)
chunks = [
    TextChunk(content=restore_scripture_references(c.content, refs), **c_meta)
    for c in chunks
]

Choosing parameters

Chuukese text is denser per-character than English — when budgeting tokens for an LLM downstream, count tokens, not characters.

Pitfalls

• max_chunk_size is in characters by default (the dataclass __len__ returns len(content)). If you need token counts, post-process with the model's tokenizer.
• Setting preserve_paragraphs=True plus a small max_chunk_size will produce oversized chunks rather than break a paragraph — verify chunk lengths if you have hard caps.
• The chunker emits chunk_ids that are not deterministic across runs (they include positional info). Don't use them as DB primary keys.
• Don't subclass — the public API is small. If you need different behavior, configure the constructor or post-process the chunk list.