Fine Tuning Guide

Fine-Tuning Guide

Overview

Fine-tuning adapts a pre-trained model to a specific task or domain by training on curated data. This skill covers when to fine-tune vs use prompt engineering, dataset preparation, parameter-efficient methods (LoRA/QLoRA), alignment techniques (RLHF/DPO), and practical training with Hugging Face tooling.

Fine-Tuning vs Prompt Engineering Decision

Decision Framework

Can prompt engineering achieve your quality target?
  YES -> Use prompt engineering (cheaper, faster iteration)
  NO  -> Continue

Do you have < 100 high-quality examples?
  YES -> Use few-shot prompting instead
  NO  -> Continue

Is your task highly specialized or domain-specific?
  YES -> Fine-tuning likely needed
  NO  -> Try prompt engineering harder first

Do you need consistent output format/style?
  YES -> Fine-tuning excels at this
  # ... (condensed) ...
  NO  -> Prompt engineering may suffice

Budget for training and hosting?
  LOW  -> Prompt engineering
  HIGH -> Fine-tuning is viable

Comparison Table

Dataset Preparation

Instruction Tuning Format

[
  {
    "instruction": "Summarize the following medical report in plain language for a patient.",
    "input": "MRI findings indicate a 2.3cm hyperintense lesion in the right frontal lobe...",
    "output": "Your brain scan showed a small spot (about 1 inch) in the right front part of your brain..."
  },
  {
    "instruction": "Extract all medication names and dosages from this clinical note.",
    "input": "Patient prescribed metformin 500mg BID and lisinopril 10mg daily...",
    "output": "{\"medications\": [{\"name\": \"metformin\", \"dose\": \"500mg\", \"frequency\": \"twice daily\"}, {\"name\": \"lisinopril\", \"dose\": \"10mg\", \"frequency\": \"once daily\"}]}"
  }
]

Chat Format (Preferred for Modern Models)

{
  "messages": [
    {"role": "system", "content": "You are a medical coding assistant."},
    {"role": "user", "content": "Assign ICD-10 codes to: Patient presents with acute bronchitis and type 2 diabetes."},
    {"role": "assistant", "content": "1. J20.9 - Acute bronchitis, unspecified\n2. E11.9 - Type 2 diabetes mellitus without complications"}
  ]
}

Data Quality Pipeline

import json
from dataclasses import dataclass

@dataclass
class QualityMetrics:
    total: int
    passed: int
    failed: int
    failure_reasons: dict

def validate_dataset(data: list[dict]) -> QualityMetrics:
    """Validate fine-tuning dataset quality."""
    failures = {}
    passed = 0
# ... (condensed) ...
        total=len(data),
        passed=passed,
        failed=len(data) - passed,
        failure_reasons=failures,
    )

Data Augmentation Strategies

def augment_instruction_data(
    samples: list[dict],
    client,
    augment_factor: int = 3,
) -> list[dict]:
    """Generate additional training examples from existing ones."""
    augmented = list(samples)

    for sample in samples:
        for _ in range(augment_factor):
            # Paraphrase the instruction
            new_instruction = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{
                    # ... (condensed) ...
                "input": sample["input"],
                "output": sample["output"],
            })

    return augmented

LoRA (Low-Rank Adaptation)

How LoRA Works

LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each transformer layer. Instead of updating W (d x d), it trains A (d x r) and B (r x d) where r << d.

Original: h = W * x
LoRA:     h = W * x + (B * A) * x

Where W is frozen, A and B are trainable
r (rank) is typically 8-64 (vs d = 4096+)

LoRA with PEFT

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

def setup_lora_model(
    base_model: str = "meta-llama/Llama-3.1-8B-Instruct",
    lora_r: int = 16,
    lora_alpha: int = 32,
    lora_dropout: float = 0.05,
    target_modules: list[str] = None,
):
    """Configure a model with LoRA adapters."""

    model = AutoModelForCausalLM.from_pretrained(
        base_model,
        # ... (condensed) ...
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    total = sum(p.numel() for p in model.parameters())
    print(f"Trainable: {trainable:,} / {total:,} ({trainable/total:.2%})")

    return model

LoRA Hyperparameter Guide

Rules of thumb:

• Start with r=16, alpha=32
• For simple tasks (classification): r=8 may suffice
• For complex tasks (code, creative): r=32-64
• Always target all linear layers (not just attention)

QLoRA (Quantized LoRA)

QLoRA combines 4-bit quantization of the base model with LoRA, enabling fine-tuning of large models on consumer GPUs.

from transformers import BitsAndBytesConfig
import torch

def setup_qlora_model(
    base_model: str = "meta-llama/Llama-3.1-8B-Instruct",
    lora_r: int = 16,
):
    """Setup QLoRA: 4-bit base model + LoRA adapters."""

    # 4-bit quantization config
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",  # NormalFloat4
        bnb_4bit_compute_dtype=torch.bfloat16,
        # ... (condensed) ...
        task_type=TaskType.CAUSAL_LM,
    )

    model = get_peft_model(model, lora_config)
    return model

GPU Memory Requirements

Training with Hugging Face

SFTTrainer (Supervised Fine-Tuning)

from trl import SFTTrainer, SFTConfig
from transformers import AutoTokenizer
from datasets import load_dataset

def train_sft(
    model,
    tokenizer,
    train_dataset,
    output_dir: str = "./fine-tuned-model",
    num_epochs: int = 3,
    batch_size: int = 4,
    learning_rate: float = 2e-4,
):
    """Supervised fine-tuning with SFTTrainer."""
# ... (condensed) ...
    )

    trainer.train()
    trainer.save_model(output_dir)
    return trainer

Complete Training Script

from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

def main():
    # Load base model and tokenizer
    model_name = "meta-llama/Llama-3.1-8B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token

    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype="auto",
        # ... (condensed) ...
    trainer.train()
    trainer.save_model("./final-model")

if __name__ == "__main__":
    main()

RLHF and DPO

DPO (Direct Preference Optimization)

DPO is simpler than full RLHF -- it directly optimizes from preference pairs without a separate reward model.

from trl import DPOTrainer, DPOConfig

def train_dpo(
    model,
    tokenizer,
    preference_dataset,
    output_dir: str = "./dpo-model",
):
    """Train with DPO from preference pairs."""

    # Dataset format: each row has "chosen" and "rejected" responses
    # {
    #   "prompt": "Explain photosynthesis",
    #   "chosen": "Photosynthesis is the process by which plants...",
    # ... (condensed) ...
        args=training_args,
    )

    trainer.train()
    return trainer

Creating Preference Data

def create_preference_pairs(
    prompts: list[str],
    model,
    judge_client,
) -> list[dict]:
    """Generate preference pairs using an LLM judge."""
    pairs = []

    for prompt in prompts:
        # Generate two responses with different temperatures
        response_a = generate(model, prompt, temperature=0.3)
        response_b = generate(model, prompt, temperature=0.9)

        # Judge which is better
        # ... (condensed) ...
            pairs.append({"prompt": prompt, "chosen": response_a, "rejected": response_b})
        else:
            pairs.append({"prompt": prompt, "chosen": response_b, "rejected": response_a})

    return pairs

Overfitting Prevention

Strategies

1. Early stopping: Monitor validation loss, stop when it increases

from transformers import EarlyStoppingCallback

trainer = SFTTrainer(
    ...,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
)

2. Data diversity: Ensure training data covers the full distribution
3. Regularization: LoRA dropout, weight decay
4. Validation split: Always hold out 10-20% for validation

dataset = dataset.train_test_split(test_size=0.1, seed=42)
trainer = SFTTrainer(
    ...,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    args=SFTConfig(
        ...,
        eval_strategy="steps",
        eval_steps=50,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
    ),
)

Signs of Overfitting

Benchmarking Fine-Tuned Models

Automated Benchmarks

def benchmark_model(model, tokenizer, test_set: list[dict]) -> dict:
    """Run benchmarks against held-out test set."""
    results = {
        "exact_match": 0,
        "format_compliance": 0,
        "total": len(test_set),
    }

    for sample in test_set:
        prompt = tokenizer.apply_chat_template(
            sample["messages"][:-1],  # All except assistant reply
            tokenize=False,
            add_generation_prompt=True,
        )
# ... (condensed) ...

    results["exact_match_rate"] = results["exact_match"] / results["total"]
    results["format_compliance_rate"] = results["format_compliance"] / results["total"]

    return results

A/B Comparison: Base vs Fine-Tuned

def compare_models(
    base_model,
    fine_tuned_model,
    tokenizer,
    test_prompts: list[str],
    judge_client,
) -> dict:
    """Compare base and fine-tuned model outputs using LLM judge."""
    wins = {"base": 0, "fine_tuned": 0, "tie": 0}

    for prompt in test_prompts:
        base_output = generate(base_model, tokenizer, prompt)
        ft_output = generate(fine_tuned_model, tokenizer, prompt)

        # ... (condensed) ...
            wins["fine_tuned"] += 1
        else:
            wins["tie"] += 1

    return wins

OpenAI Fine-Tuning API

from openai import OpenAI

client = OpenAI()

# Upload training file
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune",
)

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18",
    # ... (condensed) ...
# Use fine-tuned model
response = client.chat.completions.create(
    model=status.fine_tuned_model,
    messages=[{"role": "user", "content": "Your prompt here"}],
)

Checklist

• Determine if fine-tuning is necessary vs prompt engineering
• Prepare and validate dataset (minimum 100 high-quality examples)
• Choose base model appropriate for task and compute budget
• Select LoRA vs QLoRA based on available GPU memory
• Configure LoRA hyperparameters (r, alpha, target modules)
• Set up experiment tracking (W&B or MLflow)
• Implement validation split and early stopping
• Train with gradient checkpointing for memory efficiency
• Benchmark fine-tuned model against base model
• Check for overfitting (train vs validation loss gap)
• Consider DPO if you have preference data
• Save and version the LoRA adapter weights

When to Use

Use this skill when:

• Designing or implementing fine tuning guide solutions
• Reviewing or improving existing fine tuning guide approaches
• Making architectural or implementation decisions about fine tuning guide
• Learning fine tuning guide patterns and best practices
• Troubleshooting fine tuning guide-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Fine Tuning Guide Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement fine tuning guide for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended fine tuning guide approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When fine tuning guide must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities