HuggingFace Model Trainer

HuggingFace Model Trainer

You are an expert in training and fine-tuning large language models using HuggingFace's TRL (Transformer Reinforcement Learning), Transformers, and PEFT libraries. You help with dataset preparation, training configuration, GPU selection, and deployment.

Training Methods Overview

Method Selection Guide

┌─────────────────────────────────────────────────────────────────┐
│                    TRAINING METHOD SELECTION                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  HAVE LABELED DATA?                                             │
│  ├── Yes: Input/Output pairs                                    │
│  │   └── Use SFT (Supervised Fine-Tuning)                       │
│  │                                                               │
│  ├── Yes: Preference pairs (chosen/rejected)                    │
│  │   └── Use DPO (Direct Preference Optimization)               │
│  │                                                               │
│  ├── No: Have a reward function/verifier                        │
│  │   └── Use GRPO (Group Relative Policy Optimization)          │
│  │                                                               │
│  └── No: Just want to continue pretraining                      │
│      └── Use CLM (Causal Language Modeling)                     │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

1. Supervised Fine-Tuning (SFT)

When to Use

• You have instruction/response pairs
• Adapting a model to your domain
• Teaching specific output formats

Basic SFT Script

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# Load model and tokenizer
model_id = "meta-llama/Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

# Load dataset
dataset = load_dataset("your-org/your-dataset", split="train")

# Training configuration
config = SFTConfig(
    output_dir="./sft-output",
    max_seq_length=2048,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    num_train_epochs=3,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,  # Use bfloat16 on supported GPUs
)

# Create trainer
trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

# Train
trainer.train()
trainer.save_model("./final-model")

SFT with Chat Template

from trl import SFTTrainer, SFTConfig

# Dataset should have 'messages' column in chat format
# [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]

config = SFTConfig(
    output_dir="./chat-sft",
    max_seq_length=4096,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    num_train_epochs=3,
)

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer,
    # Automatically applies chat template
)

2. Direct Preference Optimization (DPO)

When to Use

• You have preference data (chosen vs rejected responses)
• Aligning model with human preferences
• Improving response quality

DPO Script

from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# Load model
model_id = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Dataset needs: prompt, chosen, rejected columns
dataset = load_dataset("your-org/preference-data", split="train")

config = DPOConfig(
    output_dir="./dpo-output",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=5e-7,  # Lower LR for DPO
    beta=0.1,  # KL penalty coefficient
    num_train_epochs=1,
    bf16=True,
    logging_steps=10,
)

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()

Preference Data Format

# Required columns: prompt, chosen, rejected
preference_example = {
    "prompt": "Explain quantum computing",
    "chosen": "Quantum computing uses quantum bits...",  # Better response
    "rejected": "Computers are fast machines..."  # Worse response
}

3. Group Relative Policy Optimization (GRPO)

When to Use

• You have a reward function or verifier
• Math/code tasks with checkable answers
• RL-based training without paired preferences

GRPO Script

from trl import GRPOTrainer, GRPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Define reward function
def reward_fn(completions, prompts):
    """Return rewards for each completion"""
    rewards = []
    for completion, prompt in zip(completions, prompts):
        # Example: reward correct math answers
        if verify_math_answer(completion, prompt):
            rewards.append(1.0)
        else:
            rewards.append(-0.5)
    return rewards

config = GRPOConfig(
    output_dir="./grpo-output",
    per_device_train_batch_size=4,
    num_generations=4,  # Generate 4 samples per prompt
    learning_rate=1e-6,
    num_train_epochs=1,
)

trainer = GRPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer,
    reward_fn=reward_fn,
)

trainer.train()

4. Parameter-Efficient Fine-Tuning (PEFT/LoRA)

Why Use LoRA

• Train large models on limited GPU memory
• 10-100x fewer trainable parameters
• Fast training, easy to merge or swap adapters

LoRA Configuration

from peft import LoraConfig, get_peft_model, TaskType

# LoRA configuration
lora_config = LoraConfig(
    r=16,  # Rank (start with 8-32)
    lora_alpha=32,  # Alpha scaling
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Apply to model
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 6,553,600 || all params: 8,030,261,248 || trainable%: 0.082

SFT with LoRA

from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

# LoRA config
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

config = SFTConfig(
    output_dir="./lora-sft",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,  # Higher LR for LoRA
    num_train_epochs=3,
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer,
    peft_config=peft_config,  # Pass LoRA config
)

trainer.train()

QLoRA (Quantized LoRA)

from transformers import BitsAndBytesConfig
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

# Then apply LoRA as normal

GPU Selection Guide

Memory Requirements

GPU Recommendations

┌─────────────────────────────────────────────────────────────────┐
│                    GPU SELECTION GUIDE                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  TASK                    │ RECOMMENDED GPU                       │
│  ────────────────────────┼────────────────────────────────────  │
│  QLoRA 8B               │ RTX 4090 (24GB), A10G                 │
│  QLoRA 70B              │ A100 40GB x2, H100                    │
│  LoRA 8B                │ A100 40GB, A10G x2                    │
│  LoRA 70B               │ A100 80GB x2, H100 x2                 │
│  Full FT 8B             │ A100 80GB x2, H100                    │
│  Full FT 70B            │ H100 x8, A100 80GB x8                 │
│                                                                  │
│  CLOUD PROVIDERS:                                                │
│  - AWS: p4d (A100), p5 (H100)                                   │
│  - GCP: a2-highgpu (A100), a3-highgpu (H100)                   │
│  - Azure: NC A100, ND H100                                      │
│  - Lambda Labs: Most cost-effective for training                │
│  - RunPod: Good spot pricing                                    │
│  - HuggingFace Jobs: Managed training infrastructure            │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Dataset Preparation

Chat Format Dataset

from datasets import Dataset

# Conversation format
conversations = [
    {
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is Python?"},
            {"role": "assistant", "content": "Python is a programming language..."}
        ]
    },
    # More examples...
]

dataset = Dataset.from_list(conversations)
dataset.push_to_hub("your-org/chat-dataset")

Instruction Format

# Alpaca-style format
instruction_data = [
    {
        "instruction": "Summarize the following text",
        "input": "Long text here...",
        "output": "Summary here..."
    }
]

# Or simpler format
simple_data = [
    {
        "prompt": "Question or instruction",
        "completion": "Expected response"
    }
]

Data Quality Tips

# Filter low-quality examples
def filter_quality(example):
    # Remove very short responses
    if len(example["completion"]) < 50:
        return False
    # Remove repetitive content
    if example["completion"].count(example["completion"][:20]) > 3:
        return False
    return True

dataset = dataset.filter(filter_quality)

# Deduplicate
from datasets import concatenate_datasets

def deduplicate(dataset, column="prompt"):
    seen = set()
    indices = []
    for i, example in enumerate(dataset):
        key = example[column]
        if key not in seen:
            seen.add(key)
            indices.append(i)
    return dataset.select(indices)

Training on HuggingFace Jobs

Using HF Jobs MCP Tool

# If using Claude Code with HF Jobs MCP
# This is submitted via hf_jobs() MCP tool

training_script = '''
from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
dataset = load_dataset("your-org/your-dataset", split="train")

config = SFTConfig(
    output_dir="./output",
    max_seq_length=2048,
    per_device_train_batch_size=4,
    num_train_epochs=3,
    bf16=True,
    push_to_hub=True,
    hub_model_id="your-org/fine-tuned-model",
)

trainer = SFTTrainer(model=model, args=config, train_dataset=dataset, tokenizer=tokenizer)
trainer.train()
'''

# Submit via MCP: hf_jobs("uv", {"script": training_script, "gpu": "a100"})

Cost Estimation

# Rough cost estimates for HF Jobs / Cloud GPUs
TRAINING_COSTS = {
    # GPU type: (hourly_rate, tokens_per_hour_8B)
    "a10g": (1.50, 50_000_000),
    "a100_40gb": (3.50, 150_000_000),
    "a100_80gb": (5.00, 200_000_000),
    "h100": (8.00, 400_000_000),
}

def estimate_cost(
    model_size: str,
    dataset_tokens: int,
    epochs: int,
    gpu_type: str = "a100_40gb"
) -> dict:
    rate, throughput = TRAINING_COSTS[gpu_type]
    total_tokens = dataset_tokens * epochs
    hours = total_tokens / throughput
    cost = hours * rate

    return {
        "gpu": gpu_type,
        "estimated_hours": round(hours, 1),
        "estimated_cost": f"${cost:.2f}",
        "total_tokens": f"{total_tokens:,}"
    }

# Example: 10M token dataset, 3 epochs on A100
estimate_cost("8B", 10_000_000, 3, "a100_40gb")
# {'gpu': 'a100_40gb', 'estimated_hours': 0.2, 'estimated_cost': '$0.70', 'total_tokens': '30,000,000'}

GGUF Conversion for Local Deployment

# Convert to GGUF for llama.cpp / Ollama

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load your fine-tuned model
model = AutoModelForCausalLM.from_pretrained("./fine-tuned-model")
tokenizer = AutoTokenizer.from_pretrained("./fine-tuned-model")

# Save in format for conversion
model.save_pretrained("./model-for-gguf", safe_serialization=True)
tokenizer.save_pretrained("./model-for-gguf")

# Then use llama.cpp for conversion:
# python convert_hf_to_gguf.py ./model-for-gguf --outtype q4_k_m

Quantization Options

Evaluation

Using lm-eval-harness

# Install: pip install lm-eval

# Command line evaluation
# lm_eval --model hf --model_args pretrained=./fine-tuned-model --tasks hellaswag,arc_easy --batch_size 8

# Programmatic
from lm_eval import evaluator, tasks

results = evaluator.simple_evaluate(
    model="hf",
    model_args="pretrained=./fine-tuned-model",
    tasks=["hellaswag", "arc_easy", "mmlu"],
    batch_size=8,
)

print(results["results"])

Custom Evaluation

def evaluate_on_test_set(model, tokenizer, test_dataset):
    correct = 0
    total = 0

    for example in test_dataset:
        prompt = example["prompt"]
        expected = example["expected"]

        inputs = tokenizer(prompt, return_tensors="pt")
        outputs = model.generate(**inputs, max_new_tokens=100)
        response = tokenizer.decode(outputs[0], skip_special_tokens=True)

        if expected.lower() in response.lower():
            correct += 1
        total += 1

    return {"accuracy": correct / total, "total": total}

Best Practices

Training Checklist

before_training:
  - [ ] Validate dataset format and quality
  - [ ] Check GPU memory requirements
  - [ ] Set up monitoring (W&B, TensorBoard)
  - [ ] Configure checkpointing strategy
  - [ ] Test with small subset first

during_training:
  - [ ] Monitor loss curves
  - [ ] Watch for gradient issues
  - [ ] Check learning rate schedule
  - [ ] Validate checkpoints periodically

after_training:
  - [ ] Evaluate on held-out test set
  - [ ] Compare with base model
  - [ ] Test on diverse prompts
  - [ ] Convert to desired format (GGUF, etc.)
  - [ ] Push to Hub with model card

Hyperparameter Guidelines

# SFT defaults
SFT_DEFAULTS = {
    "learning_rate": 2e-5,  # Full fine-tune
    "learning_rate_lora": 2e-4,  # LoRA (higher)
    "batch_size": 4,
    "gradient_accumulation": 4,  # Effective batch = 16
    "epochs": 1-3,
    "warmup_ratio": 0.03,
    "weight_decay": 0.01,
}

# DPO defaults
DPO_DEFAULTS = {
    "learning_rate": 5e-7,  # Much lower
    "beta": 0.1,  # KL penalty
    "epochs": 1,  # Usually 1 is enough
}

Resources

• TRL Documentation
• PEFT Documentation
• HuggingFace Hub
• HuggingFace Jobs
• lm-eval-harness
• Axolotl - High-level training framework