mkurman Overview
TRL (Transformer Reinforcement Learning) is Hugging Face's library for RLHF — SFT, reward modeling, PPO, DPO, KTO, and GRPO. It's the standard post-training toolkit for aligning language models with human preferences.
Installation
uv pip install trl
SFT
from trl import SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
trainer = SFTTrainer(
model=model, tokenizer=tokenizer,
train_dataset=dataset,
args=dict(per_device_train_batch_size=4, learning_rate=2e-5, max_seq_length=2048),
)
trainer.train()
DPO
from trl import DPOTrainer
dpo = DPOTrainer(
model=model, ref_model=ref_model, tokenizer=tokenizer,
train_dataset=preference_dataset,
args=dict(per_device_train_batch_size=4, max_length=2048),
)
dpo.train()