Fine-Tuning Strategies

Fine-Tuning Strategies for LLMs

Classification

• Domain: Computer Science, AI/ML
• Category: Transfer Learning & Model Adaptation
• Novelty: 7/10 (rapidly evolving field, new methods emerging)
• Practitioner Evidence: 10/10 (industry standard, validated at scale)

Mental Model

Fine-tuning adapts a pre-trained foundation model to specific tasks/domains by continuing training on targeted data. Like hiring an experienced generalist and providing domain-specific training—the model retains broad knowledge while gaining specialized expertise. Parameter-efficient methods (LoRA, adapters) train tiny modules instead of all weights, like teaching shortcuts rather than rewriting the entire manual.

When to Use

• Pre-trained model exists but lacks domain-specific knowledge (medical, legal, code)
• Behavior modification needed (instruction-following, safety alignment, style matching)
• Task performance insufficient with prompting alone (complex reasoning, low-resource languages)
• Cost/latency constraints favor smaller specialized model over large general model
• Data privacy requires on-premise model (can't use APIs for sensitive data)

Core Framework

1. Fine-Tuning Method Selection

Choose appropriate strategy based on resources and requirements

Full Fine-Tuning:

• Update all model parameters during training
• Highest quality, requires most compute (100% parameter updates)
• Use when: Best possible accuracy required, sufficient compute available (multi-GPU)
• Memory: ~4x model size (model + gradients + optimizer states + activations)

Parameter-Efficient Fine-Tuning (PEFT):

• Train small subset of parameters (adapters, LoRA, prefix tuning)
• 50-70% cost reduction vs. full fine-tuning, near-equivalent accuracy
• Use when: Limited GPU memory, need multiple task-specific versions, fast iteration
• Memory: ~1.2x model size (base model frozen, train tiny modules)

Feature Extraction (transfer learning baseline):

• Freeze all layers except output head, train only final classifier
• Fastest, cheapest, lowest quality for complex tasks
• Use when: Dataset very small (<1K examples), highly related to pre-training task

2. LoRA (Low-Rank Adaptation)

Most popular PEFT method - inject trainable rank decomposition matrices

How LoRA Works:

• Freeze pre-trained weights W, add trainable matrices A and B: W + AB
• A is (d × r), B is (r × d) where r << d (rank 4-64 typical)
• Only train A, B (0.1-1% of parameters), merge back into W after training

LoRA Configuration:

• Rank (r): Higher = more capacity but more parameters (4-8 for simple, 16-64 for complex)
• Alpha: Scaling factor for LoRA updates (typically alpha = 2r or r)
• Target modules: Apply to query/value projections (QV) or all linear layers (QKVO)
• Dropout: 0.05-0.1 on LoRA layers to prevent overfitting

LoRA Variants:

• QLoRA: Quantize base model to 4-bit (NF4), train LoRA adapters (75% memory reduction)
• DoRA: Weight-decomposed LoRA for better convergence
• AdaLoRA: Adaptive rank allocation across layers based on importance

3. Adapter Methods

Insert small trainable modules between frozen transformer layers

Bottleneck Adapters:

• Add down-projection (d → r) → activation → up-projection (r → d) after each layer
• Typical bottleneck size: 64-256 dimensions (vs. 4096+ model hidden size)
• 2-5% additional parameters, 30-50% cost reduction vs. full fine-tuning

Prefix Tuning:

• Prepend trainable continuous vectors to key/value in each attention layer
• Prefix length: 10-50 tokens worth of virtual "instructions"
• Use when: Few-shot learning, want to condition model without changing weights

IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations):

• Learn multiplicative scaling vectors for activations (even smaller than LoRA)
• 0.01% parameters, competitive with LoRA on many tasks

4. Data Preparation & Quality

Prepare high-quality training data for effective fine-tuning

Data Volume Guidelines:

• Instruction tuning: 2K-10K diverse examples minimum
• Domain adaptation: 10K-100K domain-specific documents
• Task-specific: 500-5K task examples (depends on task complexity)
• Quality > quantity: 1K high-quality > 10K noisy examples

Data Format:

• Instruction-following: (instruction, input, output) triplets
• Conversational: Multi-turn dialogues with system/user/assistant roles
• Domain text: Unstructured documents for continued pre-training
• Ensure format matches target use case (not just Q&A if building chatbot)

Data Quality Checklist:

• Diverse coverage of target task variations
• High-quality human-written or carefully filtered outputs
• Balanced representation (avoid demographic/topic biases)
• Decontaminated (remove benchmark test sets from training data)

5. Training Configuration

Set hyperparameters for stable, effective fine-tuning

Learning Rate:

• Full fine-tuning: 1e-5 to 5e-5 (much smaller than pre-training)
• LoRA/PEFT: 1e-4 to 3e-4 (can be higher since fewer parameters)
• Use warmup: 3-10% of steps for gradual ramp-up
• Scheduler: Linear decay or cosine decay to 0

Batch Size & Gradient Accumulation:

• Effective batch size: 32-128 for most tasks (instruction tuning)
• Use gradient accumulation if GPU memory limited (micro-batch 1-4, accumulate 8-32 steps)
• Larger batches = more stable but slower adaptation

Epochs & Early Stopping:

• 1-5 epochs typical (more = overfitting risk)
• Monitor validation loss/metrics, stop if no improvement for 2-3 evaluations
• Save checkpoints every epoch for best model selection

Regularization:

• Dropout: 0.1 on adapters/LoRA, 0.0-0.05 on full fine-tuning
• Weight decay: 0.01-0.1 (L2 regularization on trainable parameters)

6. Evaluation & Iteration

Measure fine-tuning effectiveness and iterate

Quantitative Metrics:

• Task-specific: Accuracy, F1, BLEU, ROUGE depending on task
• Perplexity: Lower = better language modeling (for domain adaptation)
• General capabilities: Test on held-out benchmarks (MMLU, GSM8K) to ensure no regression

Qualitative Evaluation:

• Manual review of 50-100 model outputs across diverse inputs
• Check for: Hallucinations, off-topic responses, style inconsistency, safety issues
• A/B test vs. base model with real users when possible

Iteration Strategy:

• Start small: 1K examples, LoRA rank 8, 1 epoch → quick baseline
• Scale up: Add data, increase rank/epochs if underfitting
• Diagnose: Overfitting (train high, val low) → reduce epochs/rank; Underfitting (both low) → add capacity/data

7. Deployment & Multi-Adapter Serving

Deploy fine-tuned models efficiently in production

Single-Task Deployment:

• Merge LoRA weights back into base model (no inference overhead)
• Quantize for deployment (GPTQ, AWQ, GGUF) to reduce memory/cost
• Serve via standard inference frameworks (vLLM, TensorRT-LLM, HuggingFace TGI)

Multi-Adapter Serving:

• Keep base model in memory, load LoRA adapters dynamically per request
• Serve 10-100+ specialized models with single base model instance
• Use adapter routing: Route requests to appropriate adapter based on task/user
• Tools: Predibase, Replicate, custom vLLM with LoRA support

Practical Application

Customer Support Chatbot (Instruction Tuning)

Problem: GPT-3.5 too generic, needs company-specific knowledge and tone Fine-Tuning Solution:

1. Collect 5K customer service conversations (historical tickets + human-written responses)
2. Format as instruction-response pairs (query, context, ideal_response)
3. Fine-tune Llama-3-8B with LoRA (rank=16, alpha=32, QV layers)
4. 3 epochs, lr=2e-4, batch=64 (8 micro-batch × 8 accumulation steps)
5. Evaluate on held-out tickets, compare response quality vs. base model Result: 35% reduction in response time, 25% increase in CSAT, 4x cheaper than GPT-4 API

Medical Report Generation (Domain Adaptation)

Problem: General LLM hallucinates medical terminology, misses critical details Fine-Tuning Solution:

1. Curate 50K radiology reports (anonymized clinical data)
2. Continued pre-training (next-token prediction on domain text) for 1 epoch
3. Then instruction fine-tune on 3K (imaging_findings → clinical_report) pairs
4. Use QLoRA (4-bit base, rank=32) to fit 70B model on single A100
5. Validate with radiologist review (accuracy, completeness, safety) Result: 90% clinician acceptance rate (vs. 60% for GPT-4), compliant with privacy requirements

Code Generation for Internal APIs (Task-Specific)

Problem: Copilot doesn't know company's internal APIs and conventions Fine-Tuning Solution:

1. Extract 20K code snippets from company repos (focus on API usage)
2. Generate (docstring → code) pairs using existing well-documented functions
3. Fine-tune CodeLlama-13B with LoRA (rank=8, QKVO layers)
4. 2 epochs, lr=1e-4, add 0.1 dropout to prevent overfitting on API patterns
5. Test on hidden internal functions, measure correctness + style adherence Result: 70% acceptance rate for suggested completions (vs. 35% for base Copilot)

Edge Cases & Nuances

Catastrophic Forgetting: Fine-tuning erases general capabilities

• Use smaller learning rate (1e-5 vs. 1e-4), fewer epochs (1-2 vs. 3-5)
• Mix general instruction data (10-20%) with domain-specific data
• Evaluate on general benchmarks (MMLU) to detect regression
• Consider multi-task fine-tuning: Train on target task + diverse auxiliary tasks

Overfitting on Small Datasets: Model memorizes training data

• Strong regularization (dropout 0.2, weight decay 0.1)
• Data augmentation: Paraphrase instructions, back-translate examples
• Use smaller model (7B instead of 70B) if dataset <5K examples
• Early stopping based on validation metrics (not training loss)

Distribution Mismatch: Training data doesn't match deployment inputs

• Collect production data samples, manually label subset for validation
• Iterative deployment: Fine-tune → deploy → collect failures → retrain
• Active learning: Identify low-confidence predictions, prioritize for labeling

Adapter Interference: Multiple LoRA adapters conflict when combined

• Composition methods: Sequential (adapter1 → adapter2), merged (weighted average)
• Orthogonalization techniques to reduce interference between adapters
• Alternatively: Train multi-task adapter from scratch instead of composing

Anti-Patterns

Fine-Tuning When Prompting Sufficient: Wasting resources when few-shot prompting works Using Tiny Datasets: Attempting fine-tuning with <500 examples (prompt engineering better) No Validation Set: Overfitting without realizing, no way to select best checkpoint Copying Benchmark Data: Training on test sets, inflated metrics, poor generalization

Trade-offs

Full Fine-Tuning vs. LoRA:

• Full: Highest quality (+2-5% on benchmarks), 10x compute cost, single specialized model
• LoRA: 95% of full quality, 10% compute cost, can serve many adapters simultaneously

LoRA Rank Selection:

• Low rank (4-8): Faster, less overfitting, sufficient for simple tasks
• High rank (32-64): More capacity, better for complex tasks, higher memory/compute

Training Duration:

• 1 epoch: Fast, less overfitting, may underfit complex tasks
• 3-5 epochs: Better fit, overfitting risk, diminishing returns after 3

Related Frameworks

• Prompt Engineering: Zero-shot alternative to fine-tuning (try first)
• RAG (Retrieval-Augmented Generation): Inject knowledge without training (complementary)
• Distillation: Compress fine-tuned large model into smaller model
• RLHF (Reinforcement Learning from Human Feedback): Align model to human preferences
• Continued Pre-training: Further pre-train on domain corpus before task fine-tuning

Practitioner Sources

• Chip Huyen - AI Engineering: Fine-tuning in production, best practices, cost analysis
• HuggingFace PEFT Library: LoRA, adapters, prefix tuning implementations
• Databricks LoRA Guide: Optimal parameter selection, efficiency benchmarks
• Google ML Design Patterns: Transfer learning patterns, feature extraction strategies
• Predibase Blog: Multi-adapter serving, LoRA in production at scale
• Microsoft DeepSpeed: Memory-efficient training, ZeRO optimization for fine-tuning