Tokenization Debugger

Help debug LLM behaviors by analyzing tokenization - the "atoms" of language models that explain many "weird" behaviors.

sethmblack Updated 9 repo stars

File contents

sethmblack/paks-skills commit b954e60c5f

Frequently asked questions

npx get-skill add sethmblack/tokenization-debugger