Tokenization

Understand how text becomes tokens, and why token boundaries explain model behaviour on numbers, code, and non-English text. Use when a model behaves strangely on specific strings or costs more than expected.

Amey-Thakur Updated 14 repo stars

File contents

Amey-Thakur/AI-SKILLS commit 66210b3e5f

Frequently asked questions

npx get-skill add amey-thakur/tokenization