Token Aware Chunking

How to chunk technical documentation and source code along AST/markdown boundaries calibrated strictly to tokenizer limits (tiktoken cl100k/o200k), eliminating fragmented code syntax and embedding clipping.

alivirgo Updated 3 repo stars

File contents

alivirgo/Major-AI-Skills commit 1bb48fe6c2

Frequently asked questions

npx get-skill add alivirgo/token-aware-chunking-2