Trl Fine Tuning

TRL — Transformer Reinforcement Learning Fine-Tuning

Overview

TRL (Transformer Reinforcement Learning) is the HuggingFace library for post-training language models. It provides trainers for supervised fine-tuning (SFT), preference alignment (DPO and variants), reward modeling, and online reinforcement learning (PPO, GRPO, RLOO, OnlineDPO). This skill covers the full lifecycle from base model to human-aligned model, with copy-pasteable commands and progressive disclosure into reference files for advanced topics.

When to Use

Use this skill when any of the following apply:

• You need to instruction-tune a base model with prompt-completion pairs → SFT
• You have preference data (chosen/rejected pairs) and want alignment without a reward model → DPO
• You are building a full RLHF pipeline (SFT → Reward Model → PPO)
• You need to train a reward model to score generations
• You want online RL with a custom reward function, especially under memory constraints → GRPO
• You need PPO, RLOO, or OnlineDPO for maximum control over reinforcement learning

Trigger keywords: fine-tune, fine-tuning, SFT, DPO, PPO, GRPO, RLHF, reward model, preference alignment, post-training, instruction tuning, RL, reinforcement learning, TRL.

Use alternatives instead:

• HuggingFace Trainer — basic fine-tuning without RL or preferences
• Axolotl — YAML-based training configuration workflows
• Unsloth — fast LoRA training for single-GPU scenarios
• LitGPT — educational / minimal fine-tuning

Prerequisites

Installation

pip install trl transformers datasets peft accelerate

For GPU support ensure torch is installed with the correct CUDA build for your platform.

Hardware Requirements

• GPU: NVIDIA with CUDA required for all methods.
• Multi-GPU: Supported via accelerate. Use accelerate launch for distributed training.
• Mixed precision: BF16 recommended on A100/H100. Use FP16 on older architectures (V100, T4).
• Memory optimization: Use LoRA/QLoRA for all methods, enable gradient checkpointing, reduce batch size with gradient accumulation to maintain effective batch.

Windows Host Notes

Primary development environment is Windows with PowerShell. When running CLI commands on Windows:

• Use PowerShell syntax (backticks for line continuation, or keep commands on one line).
• Paths use backslashes in PowerShell: ~\models\Qwen2.5-0.5B-SFT.
• Python virtual environments: .\venv\Scripts\Activate.ps1.
• For accelerate launch, use accelerate launch directly in PowerShell — no special quoting needed.

Procedure

Method Selection Guide

Workflow 1: Supervised Fine-Tuning (SFT)

Instruction-tune a base model on prompt-completion pairs.

Step 1 — Load model, tokenizer, and dataset:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

dataset = load_dataset("trl-lib/Capybara", split="train")

Step 2 — Configure and train:

training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer
)
trainer.train()
trainer.save_model()

Load references/sft-training.md when you need: dataset format details, chat template configuration, packing strategies, multi-GPU training setup, or LoRA/QLoRA configuration for SFT.

Workflow 2: DPO — Preference Alignment Without a Reward Model

Align a model with chosen/rejected preference pairs. No reward model required.

Step 1 — Prepare preference dataset:

Required format (JSON):

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I don't know."
}

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# Or load your own:
# dataset = load_dataset("json", data_files="preferences.json")

Step 2 — Configure DPO:

from trl import DPOConfig

config = DPOConfig(
    output_dir="Qwen2.5-0.5B-DPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=5e-7,
    beta=0.1,               # KL penalty strength (default 0.1)
    max_prompt_length=512,
    max_length=1024,
    logging_steps=10
)

Step 3 — Train with DPOTrainer:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer
)
trainer.train()
trainer.save_model()

CLI alternative:

trl dpo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-7 \
    --beta 0.1

Load references/dpo-variants.md when you need: IPO, cDPO, RPO, or other DPO loss functions, or recommended hyperparameters for DPO variants.

Workflow 3: Full RLHF Pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Checklist:

RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model

Step 1 — SFT: Follow Workflow 1 above. Save to Qwen2.5-0.5B-SFT.

Step 2 — Train reward model:

from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig
from datasets import load_dataset

model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-SFT",
    num_labels=1  # Single reward score
)
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

training_args = RewardConfig(
    output_dir="Qwen2.5-0.5B-Reward",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=1e-5
)

trainer = RewardTrainer(
    model=model,
    args=training_args,
    processing_class=tokenizer,
    train_dataset=dataset
)
trainer.train()
trainer.save_model()

Load references/reward-modeling.md when you need: outcome vs process reward models, Bradley-Terry loss details, reward model evaluation metrics, or reward hacking mitigation.

Step 3 — PPO reinforcement learning:

python -m trl.scripts.ppo \
    --model_name_or_path Qwen2.5-0.5B-SFT \
    --reward_model_path Qwen2.5-0.5B-Reward \
    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
    --output_dir Qwen2.5-0.5B-PPO \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 64 \
    --total_episodes 10000

Load references/online-rl.md when you need: PPO, RLOO, or OnlineDPO detailed configurations, KL coefficient tuning, or multi-GPU online RL setup.

Step 4 — Evaluate:

from transformers import pipeline

generator = pipeline("text-generation", model="Qwen2.5-0.5B-PPO")

prompt = "Explain quantum computing to a 10-year-old"
output = generator(prompt, max_length=200)[0]["generated_text"]
print(output)

Workflow 4: GRPO — Memory-Efficient Online RL

Train with reinforcement learning using a custom reward function. GRPO is more memory-efficient than PPO because it does not require a separate value model.

Load references/grpo-training.md when you need: reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, or troubleshooting. A production-ready training script is in templates/basic_grpo_training.py.

Step 1 — Define reward function:

def reward_function(completions, **kwargs):
    """
    Compute rewards for completions.

    Args:
        completions: List of generated texts

    Returns:
        List of reward scores (floats)
    """
    rewards = []
    for completion in completions:
        score = len(completion.split())                    # Favor longer responses
        score += len(set(completion.lower().split()))      # Reward unique words
        rewards.append(score)
    return rewards

Or use a trained reward model:

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):
    full_texts = [p + c for p, c in zip(prompts, completions)]
    results = reward_model(full_texts)
    return [r["score"] for r in results]

Step 2 — Configure GRPO:

from trl import GRPOConfig

config = GRPOConfig(
    output_dir="Qwen2-GRPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=1e-5,
    num_generations=4,       # Generate 4 completions per prompt
    max_new_tokens=128
)

Step 3 — Train with GRPOTrainer:

from datasets import load_dataset
from trl import GRPOTrainer

dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_function,
    args=config,
    train_dataset=dataset
)
trainer.train()

CLI:

trl grpo \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --dataset_name trl-lib/tldr \
    --output_dir Qwen2-GRPO \
    --num_generations 4

Pitfalls

OOM during DPO training

DPO stores both the policy and a frozen reference model in memory. Reduce batch size and sequence length, or enable gradient checkpointing:

config = DPOConfig(
    per_device_train_batch_size=1,       # Reduce from 4
    max_length=512,                      # Reduce from 1024
    gradient_accumulation_steps=8        # Maintain effective batch
)

model.gradient_checkpointing_enable()

Poor DPO alignment quality

The beta parameter controls the KL penalty strength. Tune it:

• Higher beta (e.g. 0.5) = more conservative, stays closer to reference model.
• Lower beta (e.g. 0.01) = more aggressive alignment, risk of over-optimization.

config = DPOConfig(beta=0.5)   # More conservative
config = DPOConfig(beta=0.01)  # More aggressive

Reward model not learning

Check loss type and learning rate. Reward models often need lower LR and more epochs:

config = RewardConfig(
    learning_rate=1e-5,    # Try different LR
    num_train_epochs=3     # Train longer
)

Verify your preference dataset has clear winners:

print(dataset[0])
# Should show clear chosen > rejected quality difference

PPO training unstable

Adjust KL coefficient and clip range:

config = PPOConfig(
    kl_coef=0.1,      # Increase from 0.05 to stabilize
    cliprange=0.1     # Reduce from 0.2 to limit policy updates
)

GRPO loss increasing

GRPO loss can increase during training — this is expected behavior and does not necessarily indicate a problem. The loss in RL is not directly comparable to supervised losses. See references/grpo-training.md for detailed explanation and mode collapse detection.

General memory optimization

• Use LoRA/QLoRA for all methods to drastically reduce VRAM.
• Enable gradient checkpointing: model.gradient_checkpointing_enable().
• Reduce per_device_train_batch_size and compensate with gradient_accumulation_steps.
• Use BF16 mixed precision on A100/H100.

Verification

Verify SFT model was saved

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen2.5-0.5B-SFT")
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")
print("Model loaded successfully:", model.config.model_type)

Verify DPO model generates aligned outputs

from transformers import pipeline

generator = pipeline("text-generation", model="Qwen2.5-0.5B-DPO", tokenizer="Qwen2.5-0.5B-DPO")
output = generator("What is the capital of France?", max_new_tokens=50)[0]["generated_text"]
print(output)
# Expect: response aligned with preference data style

Verify reward model produces scores

from transformers import pipeline

reward_pipe = pipeline("text-classification", model="Qwen2.5-0.5B-Reward")
score = reward_pipe("The capital of France is Paris.")
print("Reward score:", score)
# Expect: a float score in the "score" field

Verify training logs

Check that loss is decreasing (SFT, DPO, Reward Model) or that reward is increasing (PPO, GRPO):

# Check trainer_state.json in output directory
cat Qwen2.5-0.5B-SFT/trainer_state.json | python -m json.tool | head -20

On Windows PowerShell:

Get-Content Qwen2.5-0.5B-SFT\trainer_state.json | ConvertFrom-Json | Select-Object -ExpandProperty log_history | Select-Object -First 5

Verify GRPO reward trend

import json

with open("Qwen2-GRPO/trainer_state.json") as f:
    state = json.load(f)

rewards = [entry.get("reward") for entry in state["log_history"] if "reward" in entry]
print("Reward trend:", rewards)
# Expect: generally increasing reward over training steps

Related Skills

• peft-lora — LoRA/QLoRA configuration for memory-efficient fine-tuning
• huggingface-training — Base HuggingFace Trainer for non-RL fine-tuning
• axolotl-training — YAML-based training configuration alternative
• unsloth-finetuning — Fast LoRA training for single-GPU scenarios

Resources

• TRL Docs: https://huggingface.co/docs/trl/
• TRL GitHub: https://github.com/huggingface/trl
• Example scripts: https://github.com/huggingface/trl/tree/main/examples/scripts
• Key papers:
  • "Training language models to follow instructions with human feedback" (InstructGPT, 2022)
  • "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)
  • "Group Relative Policy Optimization" (GRPO, 2024)

Reference Files

Load these when the corresponding topic is needed — do not load all at once: