LLM Pre-training Helper
A skill for training language models from scratch using PyTorch, following best practices from the "LLMs from Scratch" methodology.
What this skill does
This skill helps you:
- Set up GPT model architectures with proper configuration
- Prepare training data with tokenization and data loaders
- Configure training loops with loss monitoring and evaluation
- Implement text generation with sampling strategies
- Save and load model checkpoints
- Visualize training progress (loss, perplexity)
When to use this skill
Use this skill when:
- You want to train an LLM from scratch on your own dataset
- You need to understand the pre-training workflow
- You're setting up GPT model configurations
- You want to monitor training metrics (loss, perplexity)
- You need to save/load model checkpoints
- You're implementing text generation with temperature/top-k sampling
Quick Start
1. Set up model configuration
GPT_CONFIG = {
"vocab_size": 50257, # GPT-2 vocabulary size
"context_length": 256, # Context window (adjust based on data)
"emb_dim": 768, # Embedding dimension
"n_heads": 12, # Attention heads
"n_layers": 12, # Transformer layers
"drop_rate": 0.1, # Dropout rate
"qkv_bias": False # Query-key-value bias
}
2. Prepare your data
# Load your text data
text_data = "your training text here"
# Split into train/validation (90/10 is common)
train_ratio = 0.90
split_idx = int(train_ratio * len(text_data))
train_data = text_data[:split_idx]
val_data = text_data[split_idx:]
# Create data loaders
train_loader = create_dataloader_v1(
train_data,
batch_size=2,
max_length=GPT_CONFIG["context_length"],
stride=GPT_CONFIG["context_length"],
shuffle=True,
drop_last=True
)
val_loader = create_dataloader_v1(
val_data,
batch_size=2,
max_length=GPT_CONFIG["context_length"],
stride=GPT_CONFIG["context_length"],
shuffle=False,
drop_last=False
)
3. Initialize model and start training
import torch
# Set seed for reproducibility
torch.manual_seed(123)
# Initialize model
model = GPTModel(GPT_CONFIG)
# Select device
if torch.cuda.is_available():
device = torch.device("cuda")
elif torch.backends.mps.is_available():
device = torch.device("mps")
else:
device = torch.device("cpu")
model.to(device)
# Setup optimizer
optimizer = torch.optim.AdamW(
model.parameters(),
lr=0.0004,
weight_decay=0.1
)
# Train
num_epochs = 10
train_losses, val_losses, tokens_seen = train_model_simple(
model, train_loader, val_loader, optimizer, device,
num_epochs=num_epochs,
eval_freq=5, # Evaluate every 5 steps
eval_iter=5, # Use 5 batches for evaluation
start_context="Your starting phrase",
tokenizer=tokenizer
)
Core Components
Model Architecture
The GPT model consists of:
- Token embeddings: Convert token IDs to vectors
- Positional embeddings: Add position information
- Transformer blocks: Multi-head attention + feed-forward
- Output head: Maps embeddings back to vocabulary
Training Loop Structure
For each epoch:
For each batch:
1. Zero gradients
2. Forward pass → get logits
3. Calculate loss (cross-entropy)
4. Backward pass → compute gradients
5. Optimizer step → update weights
6. (Optional) Evaluate and log metrics
Loss Functions
- Cross-entropy loss: Measures difference between predicted and actual token distributions
- Perplexity:
exp(loss) - represents model uncertainty (lower is better)
Text Generation Strategies
| Strategy |
Description |
Use Case |
| Greedy |
Always pick highest probability token |
Deterministic output |
| Top-k |
Sample from top k tokens |
Balanced diversity |
| Temperature |
Scale logits before softmax |
Control randomness |
| Top-p (nucleus) |
Sample until cumulative probability threshold |
Adaptive diversity |
Training Parameters Guide
Learning Rate
- Small (1e-5 to 1e-4): Precise convergence, slower training
- Large (1e-3 to 1e-2): Faster training, risk of overshooting
- Recommended: Start with 4e-4 for AdamW
Batch Size
- Small (1-4): More frequent updates, noisier gradients
- Large (8-32): Smoother gradients, more memory
- Recommended: 2-4 for CPU, 8-16 for GPU
Context Length
- Short (128-256): Faster training, less context
- Long (512-1024): More context, slower training
- Recommended: Match your use case, start with 256
Number of Epochs
- Few (5-10): Quick iteration, may underfit
- Many (20-50): Better convergence, risk of overfitting
- Recommended: Monitor validation loss, stop when it plateaus
Monitoring Training
Key Metrics to Track
- Training Loss: Should decrease over time
- Validation Loss: Should decrease, watch for overfitting
- Perplexity:
exp(loss), lower is better
- Tokens Seen: Track progress through dataset
Signs of Good Training
- Training loss steadily decreases
- Validation loss follows training loss
- Generated text becomes more coherent
- Perplexity drops significantly
Signs of Problems
- Overfitting: Training loss ↓, Validation loss ↑
- Underfitting: Both losses stay high
- Exploding gradients: Loss becomes NaN or inf
- Vanishing gradients: Loss stops decreasing
Saving and Loading Models
Save Full Checkpoint (for resuming training)
torch.save({
"model_state_dict": model.state_dict(),
"optimizer_state_dict": optimizer.state_dict(),
"epoch": current_epoch,
"loss": current_loss
}, "checkpoint.pth")
Load Full Checkpoint
checkpoint = torch.load("checkpoint.pth", map_location=device)
model.load_state_dict(checkpoint["model_state_dict"])
optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
model.train()
Save Model Only (for inference)
torch.save(model.state_dict(), "model.pth")
Load Model Only
model = GPTModel(GPT_CONFIG)
model.load_state_dict(torch.load("model.pth", map_location=device))
model.eval()
Common Issues and Solutions
"Not enough tokens for training"
- Solution: Reduce
context_length or increase training data
- Check:
total_tokens * train_ratio >= context_length
"CUDA out of memory"
- Solution: Reduce batch size or context length
- Alternative: Use gradient accumulation
"Loss not decreasing"
- Check: Learning rate (try 1e-4 to 1e-3)
- Check: Data quality and tokenization
- Check: Model is in training mode (
model.train())
"Validation loss increasing"
- Solution: Early stopping, reduce epochs
- Alternative: Add regularization (dropout, weight decay)
Advanced Techniques (Not in Base Code)
Learning Rate Scheduling
- Linear Warmup: Start small, increase to max LR
- Cosine Decay: Gradually reduce LR after warmup
Gradient Clipping
- Prevents exploding gradients
- Set
max_norm in optimizer or use torch.nn.utils.clip_grad_norm_
Top-p Sampling (Nucleus)
- More adaptive than top-k
- Sums probabilities until threshold (e.g., 0.9)
Beam Search
- Explores multiple sequences simultaneously
- Better quality than greedy, more expensive
Next Steps
After training:
- Evaluate: Test on held-out data
- Fine-tune: Adapt to specific tasks
- Deploy: Use for inference or as base model
- Iterate: Adjust hyperparameters and retrain
References
1---2name: llm-pretraining-helper3description: How to train LLMs from scratch using PyTorch, including model architecture setup, data preparation, training loops, loss monitoring, and model saving/loading. Use this skill whenever the user wants to train a language model from scratch, understand pre-training workflows, set up GPT architectures, configure training parameters, monitor loss/perplexity, or load/save model checkpoints. Make sure to use this skill when users mention training LLMs, pre-training, model checkpoints, GPT architectures, training loops, or want to build language models from the ground up.4metadata:5 source: https://skillmd.com/skills/abelrguezr/llm-pretraining-helper6---78# LLM Pre-training Helper910A skill for training language models from scratch using PyTorch, following best practices from the "LLMs from Scratch" methodology.1112## What this skill does1314This skill helps you:15- Set up GPT model architectures with proper configuration16- Prepare training data with tokenization and data loaders17- Configure training loops with loss monitoring and evaluation18- Implement text generation with sampling strategies19- Save and load model checkpoints20- Visualize training progress (loss, perplexity)2122## When to use this skill2324Use this skill when:25- You want to train an LLM from scratch on your own dataset26- You need to understand the pre-training workflow27- You're setting up GPT model configurations28- You want to monitor training metrics (loss, perplexity)29- You need to save/load model checkpoints30- You're implementing text generation with temperature/top-k sampling3132## Quick Start3334### 1. Set up model configuration3536```python37GPT_CONFIG = {38 "vocab_size": 50257, # GPT-2 vocabulary size39 "context_length": 256, # Context window (adjust based on data)40 "emb_dim": 768, # Embedding dimension41 "n_heads": 12, # Attention heads42 "n_layers": 12, # Transformer layers43 "drop_rate": 0.1, # Dropout rate44 "qkv_bias": False # Query-key-value bias45}46```4748### 2. Prepare your data4950```python51# Load your text data52text_data = "your training text here"5354# Split into train/validation (90/10 is common)55train_ratio = 0.9056split_idx = int(train_ratio * len(text_data))57train_data = text_data[:split_idx]58val_data = text_data[split_idx:]5960# Create data loaders61train_loader = create_dataloader_v1(62 train_data,63 batch_size=2,64 max_length=GPT_CONFIG["context_length"],65 stride=GPT_CONFIG["context_length"],66 shuffle=True,67 drop_last=True68)6970val_loader = create_dataloader_v1(71 val_data,72 batch_size=2,73 max_length=GPT_CONFIG["context_length"],74 stride=GPT_CONFIG["context_length"],75 shuffle=False,76 drop_last=False77)78```7980### 3. Initialize model and start training8182```python83import torch8485# Set seed for reproducibility86torch.manual_seed(123)8788# Initialize model89model = GPTModel(GPT_CONFIG)9091# Select device92if torch.cuda.is_available():93 device = torch.device("cuda")94elif torch.backends.mps.is_available():95 device = torch.device("mps")96else:97 device = torch.device("cpu")9899model.to(device)100101# Setup optimizer102optimizer = torch.optim.AdamW(103 model.parameters(),104 lr=0.0004,105 weight_decay=0.1106)107108# Train109num_epochs = 10110train_losses, val_losses, tokens_seen = train_model_simple(111 model, train_loader, val_loader, optimizer, device,112 num_epochs=num_epochs,113 eval_freq=5, # Evaluate every 5 steps114 eval_iter=5, # Use 5 batches for evaluation115 start_context="Your starting phrase",116 tokenizer=tokenizer117)118```119120## Core Components121122### Model Architecture123124The GPT model consists of:125- **Token embeddings**: Convert token IDs to vectors126- **Positional embeddings**: Add position information127- **Transformer blocks**: Multi-head attention + feed-forward128- **Output head**: Maps embeddings back to vocabulary129130### Training Loop Structure131132```133For each epoch:134 For each batch:135 1. Zero gradients136 2. Forward pass → get logits137 3. Calculate loss (cross-entropy)138 4. Backward pass → compute gradients139 5. Optimizer step → update weights140 6. (Optional) Evaluate and log metrics141```142143### Loss Functions144145- **Cross-entropy loss**: Measures difference between predicted and actual token distributions146- **Perplexity**: `exp(loss)` - represents model uncertainty (lower is better)147148### Text Generation Strategies149150| Strategy | Description | Use Case |151|----------|-------------|----------|152| Greedy | Always pick highest probability token | Deterministic output |153| Top-k | Sample from top k tokens | Balanced diversity |154| Temperature | Scale logits before softmax | Control randomness |155| Top-p (nucleus) | Sample until cumulative probability threshold | Adaptive diversity |156157## Training Parameters Guide158159### Learning Rate160- **Small (1e-5 to 1e-4)**: Precise convergence, slower training161- **Large (1e-3 to 1e-2)**: Faster training, risk of overshooting162- **Recommended**: Start with 4e-4 for AdamW163164### Batch Size165- **Small (1-4)**: More frequent updates, noisier gradients166- **Large (8-32)**: Smoother gradients, more memory167- **Recommended**: 2-4 for CPU, 8-16 for GPU168169### Context Length170- **Short (128-256)**: Faster training, less context171- **Long (512-1024)**: More context, slower training172- **Recommended**: Match your use case, start with 256173174### Number of Epochs175- **Few (5-10)**: Quick iteration, may underfit176- **Many (20-50)**: Better convergence, risk of overfitting177- **Recommended**: Monitor validation loss, stop when it plateaus178179## Monitoring Training180181### Key Metrics to Track1821831. **Training Loss**: Should decrease over time1842. **Validation Loss**: Should decrease, watch for overfitting1853. **Perplexity**: `exp(loss)`, lower is better1864. **Tokens Seen**: Track progress through dataset187188### Signs of Good Training189- Training loss steadily decreases190- Validation loss follows training loss191- Generated text becomes more coherent192- Perplexity drops significantly193194### Signs of Problems195- **Overfitting**: Training loss ↓, Validation loss ↑196- **Underfitting**: Both losses stay high197- **Exploding gradients**: Loss becomes NaN or inf198- **Vanishing gradients**: Loss stops decreasing199200## Saving and Loading Models201202### Save Full Checkpoint (for resuming training)203204```python205torch.save({206 "model_state_dict": model.state_dict(),207 "optimizer_state_dict": optimizer.state_dict(),208 "epoch": current_epoch,209 "loss": current_loss210}, "checkpoint.pth")211```212213### Load Full Checkpoint214215```python216checkpoint = torch.load("checkpoint.pth", map_location=device)217model.load_state_dict(checkpoint["model_state_dict"])218optimizer.load_state_dict(checkpoint["optimizer_state_dict"])219model.train()220```221222### Save Model Only (for inference)223224```python225torch.save(model.state_dict(), "model.pth")226```227228### Load Model Only229230```python231model = GPTModel(GPT_CONFIG)232model.load_state_dict(torch.load("model.pth", map_location=device))233model.eval()234```235236## Common Issues and Solutions237238### "Not enough tokens for training"239- **Solution**: Reduce `context_length` or increase training data240- **Check**: `total_tokens * train_ratio >= context_length`241242### "CUDA out of memory"243- **Solution**: Reduce batch size or context length244- **Alternative**: Use gradient accumulation245246### "Loss not decreasing"247- **Check**: Learning rate (try 1e-4 to 1e-3)248- **Check**: Data quality and tokenization249- **Check**: Model is in training mode (`model.train()`)250251### "Validation loss increasing"252- **Solution**: Early stopping, reduce epochs253- **Alternative**: Add regularization (dropout, weight decay)254255## Advanced Techniques (Not in Base Code)256257### Learning Rate Scheduling258- **Linear Warmup**: Start small, increase to max LR259- **Cosine Decay**: Gradually reduce LR after warmup260261### Gradient Clipping262- Prevents exploding gradients263- Set `max_norm` in optimizer or use `torch.nn.utils.clip_grad_norm_`264265### Top-p Sampling (Nucleus)266- More adaptive than top-k267- Sums probabilities until threshold (e.g., 0.9)268269### Beam Search270- Explores multiple sequences simultaneously271- Better quality than greedy, more expensive272273## Next Steps274275After training:2761. **Evaluate**: Test on held-out data2772. **Fine-tune**: Adapt to specific tasks2783. **Deploy**: Use for inference or as base model2794. **Iterate**: Adjust hyperparameters and retrain280281## References282283- [LLMs from Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch)284- [rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch)285- [GPT-2 Architecture](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf)