Text Tokenizer
A skill for tokenizing text into numerical IDs for machine learning models.
What is Tokenizing?
Tokenizing is the process of breaking down text into smaller pieces called tokens, each assigned a unique numerical ID. This is fundamental for preparing text for ML models, especially in NLP.
Goal: Divide input into tokens (IDs) in a way that makes sense for the model.
Basic Tokenization
1. Splitting Text
- Simple tokenizer splits text into words and punctuation
- Example:
"Hello, world!" → ["Hello", ",", "world", "!"]
2. Creating a Vocabulary
- Maps each token to a numerical ID
- Special tokens:
[BOS] (Beginning of Sequence): Marks text start
[EOS] (End of Sequence): Marks text end
[PAD] (Padding): Makes sequences same length in batches
[UNK] (Unknown): Represents tokens not in vocabulary
- Example:
"Hello, world!" → [64, 455, 78, 467]
3. Handling Unknown Words
- Words not in vocabulary get replaced with
[UNK]
- Example:
"Bye, world!" → [987, 455, 78, 467] (assuming [UNK] = 987)
Advanced Tokenization Methods
Byte Pair Encoding (BPE)
- Purpose: Reduces vocabulary size, handles rare/unknown words
- How it works:
- Starts with individual characters as tokens
- Iteratively merges most frequent token pairs
- Continues until no more frequent pairs exist
- Benefits:
- Eliminates need for
[UNK] token
- More efficient and flexible vocabulary
- Example:
"playing" → ["play", "ing"]
WordPiece
- Used by: BERT and similar models
- Purpose: Similar to BPE, breaks words into subword units
- How it works:
- Begins with base vocabulary of individual characters
- Iteratively adds most frequent subword that maximizes training data likelihood
- Uses probabilistic model for merging decisions
- Benefits:
- Balances vocabulary size with word representation
- Efficiently handles rare and compound words
- Example:
"unhappiness" → ["un", "happy", "ness"]
Unigram Language Model
- Used by: SentencePiece
- Purpose: Uses probabilistic model for optimal subword selection
- How it works:
- Starts with large set of potential tokens
- Iteratively removes tokens that least improve model probability
- Finalizes vocabulary with most probable subword units
- Benefits:
- Flexible and natural language modeling
- Often results in more efficient tokenizations
- Example:
"internationalization" → ["international", "ization"]
Implementation with tiktoken
Basic Usage
import tiktoken
# Load GPT-2 tokenizer
encoding = tiktoken.get_encoding("gpt2")
# Encode text to token IDs
token_ids = encoding.encode("Hello, world!")
print(token_ids) # [15496, 11, 995, 0]
# Decode token IDs back to text
text = encoding.decode(token_ids)
print(text) # "Hello, world!"
With Special Tokens
# Encode with special tokens allowed
token_ids = encoding.encode("Hello, world!", allowed_special={"[EOS]"})
# Check token count
print(len(token_ids))
Processing Files
import urllib.request
# Download text file
url = "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt"
file_path = "the-verdict.txt"
urllib.request.urlretrieve(url, file_path)
# Read and tokenize
with open(file_path, "r", encoding="utf-8") as f:
raw_text = f.read()
token_ids = tiktoken.get_encoding("gpt2").encode(raw_text, allowed_special={"[EOS]"})
# Print first 50 tokens
print(token_ids[:50])
Common Use Cases
- Preprocessing text for training - Convert training data to token IDs
- Understanding model input requirements - Know what format your model expects
- Debugging tokenization issues - Inspect how text is being tokenized
- Comparing different tokenization methods - Evaluate BPE vs WordPiece vs Unigram
Best Practices
- Choose tokenizer based on your model - GPT-2 uses BPE, BERT uses WordPiece
- Handle special tokens appropriately - Include
[BOS], [EOS] as needed for your use case
- Consider vocabulary size vs. tokenization quality tradeoff - Larger vocabularies may tokenize more efficiently but use more memory
- Test with edge cases - Rare words, special characters, different languages
- Use the right encoding - Match the tokenizer to your model architecture
Troubleshooting
Unknown tokens appearing
- Check if your vocabulary is large enough
- Consider using BPE or WordPiece instead of basic tokenization
- Verify special tokens are properly configured
Token count seems too high
- Try a different tokenization method (BPE often produces fewer tokens)
- Check if you're including unnecessary whitespace or special characters
Decoding produces unexpected output
- Ensure you're using the same encoding for encode/decode
- Check if special tokens are being handled correctly
- Verify the token IDs are valid for your vocabulary
References
1---2name: text-tokenizer3description: How to tokenize text for LLMs and NLP models. Use this skill whenever the user needs to convert text into token IDs, understand tokenization methods (BPE, WordPiece, Unigram), work with vocabularies, or implement tokenization for machine learning. Make sure to use this skill when users mention tokenizing, token IDs, vocabulary creation, BPE, WordPiece, or any text preprocessing for ML models.4metadata:5 source: https://skillmd.com/skills/abelrguezr/text-tokenizer6---78# Text Tokenizer910A skill for tokenizing text into numerical IDs for machine learning models.1112## What is Tokenizing?1314Tokenizing is the process of breaking down text into smaller pieces called tokens, each assigned a unique numerical ID. This is fundamental for preparing text for ML models, especially in NLP.1516**Goal:** Divide input into tokens (IDs) in a way that makes sense for the model.1718## Basic Tokenization1920### 1. Splitting Text21- Simple tokenizer splits text into words and punctuation22- Example: `"Hello, world!"` → `["Hello", ",", "world", "!"]`2324### 2. Creating a Vocabulary25- Maps each token to a numerical ID26- **Special tokens:**27 - `[BOS]` (Beginning of Sequence): Marks text start28 - `[EOS]` (End of Sequence): Marks text end29 - `[PAD]` (Padding): Makes sequences same length in batches30 - `[UNK]` (Unknown): Represents tokens not in vocabulary31- Example: `"Hello, world!"` → `[64, 455, 78, 467]`3233### 3. Handling Unknown Words34- Words not in vocabulary get replaced with `[UNK]`35- Example: `"Bye, world!"` → `[987, 455, 78, 467]` (assuming `[UNK]` = 987)3637## Advanced Tokenization Methods3839### Byte Pair Encoding (BPE)40- **Purpose:** Reduces vocabulary size, handles rare/unknown words41- **How it works:**42 - Starts with individual characters as tokens43 - Iteratively merges most frequent token pairs44 - Continues until no more frequent pairs exist45- **Benefits:**46 - Eliminates need for `[UNK]` token47 - More efficient and flexible vocabulary48- **Example:** `"playing"` → `["play", "ing"]`4950### WordPiece51- **Used by:** BERT and similar models52- **Purpose:** Similar to BPE, breaks words into subword units53- **How it works:**54 - Begins with base vocabulary of individual characters55 - Iteratively adds most frequent subword that maximizes training data likelihood56 - Uses probabilistic model for merging decisions57- **Benefits:**58 - Balances vocabulary size with word representation59 - Efficiently handles rare and compound words60- **Example:** `"unhappiness"` → `["un", "happy", "ness"]`6162### Unigram Language Model63- **Used by:** SentencePiece64- **Purpose:** Uses probabilistic model for optimal subword selection65- **How it works:**66 - Starts with large set of potential tokens67 - Iteratively removes tokens that least improve model probability68 - Finalizes vocabulary with most probable subword units69- **Benefits:**70 - Flexible and natural language modeling71 - Often results in more efficient tokenizations72- **Example:** `"internationalization"` → `["international", "ization"]`7374## Implementation with tiktoken7576### Basic Usage7778```python79import tiktoken8081# Load GPT-2 tokenizer82encoding = tiktoken.get_encoding("gpt2")8384# Encode text to token IDs85token_ids = encoding.encode("Hello, world!")86print(token_ids) # [15496, 11, 995, 0]8788# Decode token IDs back to text89text = encoding.decode(token_ids)90print(text) # "Hello, world!"91```9293### With Special Tokens9495```python96# Encode with special tokens allowed97token_ids = encoding.encode("Hello, world!", allowed_special={"[EOS]"})9899# Check token count100print(len(token_ids))101```102103### Processing Files104105```python106import urllib.request107108# Download text file109url = "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt"110file_path = "the-verdict.txt"111urllib.request.urlretrieve(url, file_path)112113# Read and tokenize114with open(file_path, "r", encoding="utf-8") as f:115 raw_text = f.read()116117token_ids = tiktoken.get_encoding("gpt2").encode(raw_text, allowed_special={"[EOS]"})118119# Print first 50 tokens120print(token_ids[:50])121```122123## Common Use Cases1241251. **Preprocessing text for training** - Convert training data to token IDs1262. **Understanding model input requirements** - Know what format your model expects1273. **Debugging tokenization issues** - Inspect how text is being tokenized1284. **Comparing different tokenization methods** - Evaluate BPE vs WordPiece vs Unigram129130## Best Practices131132- **Choose tokenizer based on your model** - GPT-2 uses BPE, BERT uses WordPiece133- **Handle special tokens appropriately** - Include `[BOS]`, `[EOS]` as needed for your use case134- **Consider vocabulary size vs. tokenization quality tradeoff** - Larger vocabularies may tokenize more efficiently but use more memory135- **Test with edge cases** - Rare words, special characters, different languages136- **Use the right encoding** - Match the tokenizer to your model architecture137138## Troubleshooting139140### Unknown tokens appearing141- Check if your vocabulary is large enough142- Consider using BPE or WordPiece instead of basic tokenization143- Verify special tokens are properly configured144145### Token count seems too high146- Try a different tokenization method (BPE often produces fewer tokens)147- Check if you're including unnecessary whitespace or special characters148149### Decoding produces unexpected output150- Ensure you're using the same encoding for encode/decode151- Check if special tokens are being handled correctly152- Verify the token IDs are valid for your vocabulary153154## References155156- [Build a Large Language Model from Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)157- [LLMs from Scratch - Tokenization](https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/01_main-chapter-code/ch02.ipynb)