FerroxLabs LLM Fine-Tuner
Overview
Fine-tuning large language models adapts pre-trained foundation models to specific domains, tasks, or behaviors. This skill covers the full fine-tuning workflow: deciding when to fine-tune vs prompt engineer, curating high-quality datasets, choosing between full fine-tuning and parameter-efficient methods (LoRA, QLoRA), training with modern tooling, evaluating results rigorously, and deploying fine-tuned models to production.
When to Fine-Tune vs Prompt Engineer
Decision Matrix:
Factor Prompt Engineering Fine-Tuning
------------------------------------------------------------
Latency sensitivity Low (long prompts OK) High (need short prompts)
Training data available < 100 examples 100 - 100K+ examples
Task specificity General tasks Domain-specific behavior
Output format control Moderate High (consistent format)
Cost per inference Higher (long context) Lower (learned behavior)
Iteration speed Minutes Hours to days
Knowledge freshness Retrieval/RAG based Baked into weights
Behavior modification Limited Deep behavior change
Compute budget Zero training cost GPU hours required
Decision: Fine-tune when you need consistent specialized behavior
that cannot be achieved through prompting alone, AND you have
sufficient quality training data.
Dataset Preparation
Instruction Tuning Dataset Format
# Standard instruction format (Alpaca-style)
instruction_dataset = [
{
"instruction": "Classify the sentiment of this customer review.",
"input": "The product arrived damaged and customer service was unhelpful.",
"output": "Negative. The review expresses dissatisfaction with both product quality (arrived damaged) and customer service (unhelpful)."
},
{
"instruction": "Extract all medication names from this clinical note.",
"input": "Patient is currently taking metformin 500mg BID, lisinopril 10mg daily, and was prescribed amoxicillin 500mg TID for 7 days.",
"output": "1. Metformin 500mg (twice daily)\n2. Lisinopril 10mg (once daily)\n3. Amoxicillin 500mg (three times daily, 7-day course)"
}
]
# ChatML / conversation format
chat_dataset = [
{
"messages": [
{"role": "system", "content": "You are a medical coding assistant."},
{"role": "user", "content": "Code this diagnosis: Type 2 diabetes with peripheral neuropathy"},
{"role": "assistant", "content": "ICD-10: E11.40 - Type 2 diabetes mellitus with diabetic neuropathy, unspecified"}
]
}
]
Data Quality Pipeline
import hashlib
from datasets import Dataset
class DatasetCurator:
"""Curate high-quality fine-tuning datasets."""
def curate(self, raw_data: list[dict]) -> Dataset:
data = raw_data
initial_count = len(data)
# Step 1: Deduplicate
data = self._deduplicate(data)
print(f"After dedup: {len(data)} (removed {initial_count - len(data)})")
# Step 2: Filter by quality
data = [d for d in data if self._quality_check(d)]
print(f"After quality filter: {len(data)}")
# Step 3: Filter by length
data = [d for d in data if self._length_check(d)]
print(f"After length filter: {len(data)}")
# Step 4: Balance distribution
data = self._balance_distribution(data)
print(f"After balancing: {len(data)}")
return Dataset.from_list(data)
def _deduplicate(self, data):
seen = set()
unique = []
for item in data:
content_hash = hashlib.md5(
(item.get('instruction', '') + item.get('input', '')).encode()
).hexdigest()
if content_hash not in seen:
seen.add(content_hash)
unique.append(item)
return unique
def _quality_check(self, item):
"""Filter low-quality examples."""
output = item.get('output', '')
# Reject empty or very short outputs
if len(output.strip()) < 10:
return False
# Reject outputs that are just the input repeated
if output.strip() == item.get('input', '').strip():
return False
# Reject outputs with obvious errors
if 'as an ai language model' in output.lower():
return False
return True
def _length_check(self, item, max_tokens=2048):
"""Ensure examples fit within context window."""
total_text = item.get('instruction', '') + item.get('input', '') + item.get('output', '')
# Rough token estimate: 1 token per 4 characters
estimated_tokens = len(total_text) / 4
return estimated_tokens <= max_tokens
def _balance_distribution(self, data):
"""Balance task type distribution to prevent overfitting."""
from collections import Counter
if 'task_type' not in data[0]:
return data
type_counts = Counter(d['task_type'] for d in data)
median_count = sorted(type_counts.values())[len(type_counts) // 2]
max_per_type = median_count * 3 # Cap at 3x median
balanced = []
type_tracker = Counter()
for d in data:
if type_tracker[d['task_type']] < max_per_type:
balanced.append(d)
type_tracker[d['task_type']] += 1
return balanced
LoRA (Low-Rank Adaptation)
LoRA Configuration and Training
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
# Load base model
model_name = "meta-llama/Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto",
attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
# LoRA configuration
lora_config = LoraConfig(
r=16, # Rank (4-64, higher = more capacity)
lora_alpha=32, # Scaling factor (typically 2x rank)
target_modules=[ # Which layers to adapt
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
modules_to_save=["embed_tokens", "lm_head"], # Optional: train embeddings
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 20M || all params: 8B || trainable%: 0.25%
LoRA Hyperparameter Guide
Parameter Recommended Range Notes
--------------------------------------------------------------
r (rank) 8-64 Start with 16. Higher for complex tasks.
8 for simple classification.
32-64 for code/math.
lora_alpha 2x rank Controls learning rate scaling.
alpha/r = effective multiplier.
target_modules All attention + Start with attention only, add MLP
MLP projections if undertrained. More modules = more
capacity but slower training.
lora_dropout 0.0-0.1 0.05 is safe default. Increase if
overfitting (val loss diverges).
learning_rate 1e-4 to 5e-4 Higher than full fine-tuning.
2e-4 is common starting point.
batch_size 4-32 Larger batches stabilize training.
Use gradient accumulation if GPU
memory limited.
epochs 1-5 Start with 3. Watch for overfitting
after epoch 2-3 on small datasets.
QLoRA (Quantized LoRA)
QLoRA Training Setup
from transformers import BitsAndBytesConfig
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4 quantization
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, # Nested quantization
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-70B",
quantization_config=bnb_config,
device_map="auto",
attn_implementation="flash_attention_2",
)
# Apply LoRA on quantized model
model = get_peft_model(model, lora_config)
# Memory comparison:
# Full fine-tuning 70B: ~280GB VRAM (impossible on consumer GPUs)
# LoRA 70B (bf16): ~140GB VRAM (multi-GPU needed)
# QLoRA 70B (4-bit): ~40GB VRAM (fits on 2x A100 or 1x A100-80GB)
Training with TRL (Transformer Reinforcement Learning)
Supervised Fine-Tuning
from trl import SFTTrainer, SFTConfig
training_args = SFTConfig(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size: 4 * 4 = 16
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.1,
weight_decay=0.01,
max_grad_norm=1.0,
# Precision
bf16=True,
# Logging
logging_steps=10,
eval_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100,
save_total_limit=3,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
# SFT specific
max_seq_length=2048,
packing=True, # Pack short examples together
dataset_text_field="text",
# Memory optimization
gradient_checkpointing=True,
optim="adamw_torch_fused",
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
args=training_args,
)
trainer.train()
trainer.save_model("./fine-tuned-model")
DPO (Direct Preference Optimization)
from trl import DPOTrainer, DPOConfig
# DPO dataset format: chosen vs rejected completions
dpo_dataset = [
{
"prompt": "Explain quantum entanglement simply.",
"chosen": "Quantum entanglement is like having two coins that always land on opposite sides, no matter how far apart they are. When you flip one and get heads, the other instantly becomes tails. In quantum physics, particles can become 'entangled' so measuring one instantly determines the state of the other.",
"rejected": "Quantum entanglement is a quantum mechanical phenomenon in which the quantum states of two or more objects have to be described with reference to each other, even though the individual objects may be spatially separated. This leads to correlations between observable physical properties of the systems."
}
]
dpo_config = DPOConfig(
output_dir="./dpo-results",
num_train_epochs=1,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=5e-5, # Lower LR for DPO
beta=0.1, # KL penalty coefficient
max_length=1024,
max_prompt_length=512,
bf16=True,
gradient_checkpointing=True,
logging_steps=10,
)
dpo_trainer = DPOTrainer(
model=model,
ref_model=None, # Uses implicit reference with LoRA
train_dataset=dpo_dataset,
tokenizer=tokenizer,
args=dpo_config,
)
dpo_trainer.train()
Evaluation
Comprehensive Evaluation Framework
class FineTuneEvaluator:
"""Evaluate fine-tuned model quality."""
def evaluate(self, model, tokenizer, eval_dataset) -> dict:
results = {}
# 1. Perplexity (lower is better)
results['perplexity'] = self._compute_perplexity(model, tokenizer, eval_dataset)
# 2. Task-specific accuracy
results['task_accuracy'] = self._evaluate_task(model, tokenizer, eval_dataset)
# 3. Regression check against base model
results['regression'] = self._check_regression(model, tokenizer)
# 4. Generation quality (human-evaluate proxy)
results['generation_quality'] = self._evaluate_generation(model, tokenizer, eval_dataset)
return results
def _evaluate_task(self, model, tokenizer, dataset):
"""Evaluate on held-out examples with exact/fuzzy match."""
correct = 0
total = 0
for example in dataset:
prompt = format_prompt(example)
generated = generate(model, tokenizer, prompt, max_new_tokens=256)
expected = example['output']
# Exact match
if generated.strip() == expected.strip():
correct += 1
# Fuzzy match (for free-form text)
elif self._fuzzy_match(generated, expected) > 0.8:
correct += 0.5
total += 1
return correct / total if total > 0 else 0
def _check_regression(self, model, tokenizer):
"""Ensure fine-tuning did not degrade general capabilities."""
regression_prompts = [
"What is 2 + 2?",
"Translate 'hello' to French.",
"Write a haiku about the moon.",
]
# Compare outputs to expected baselines
regressions = []
for prompt in regression_prompts:
output = generate(model, tokenizer, prompt)
if not self._basic_sanity_check(prompt, output):
regressions.append(prompt)
return {"regressions_found": len(regressions), "details": regressions}
Deployment
Merging LoRA Weights for Deployment
from peft import PeftModel
# Load base model and LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(base_model, "./fine-tuned-lora-adapter")
# Merge LoRA weights into base model
merged_model = model.merge_and_unload()
# Save merged model
merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")
# Convert to GGUF for llama.cpp / Ollama deployment
# python convert_hf_to_gguf.py ./merged-model --outtype q4_k_m
vLLM Deployment
from vllm import LLM, SamplingParams
# Deploy merged fine-tuned model
llm = LLM(
model="./merged-model",
dtype="auto",
tensor_parallel_size=1, # Increase for multi-GPU
max_model_len=4096,
gpu_memory_utilization=0.9,
)
sampling_params = SamplingParams(
temperature=0.1, # Low for consistent domain tasks
top_p=0.95,
max_tokens=512,
stop=["<|eot_id|>"],
)
outputs = llm.generate(["Your prompt here"], sampling_params)
Fine-Tuning Checklist
Planning:
[ ] Confirmed fine-tuning is needed (not achievable via prompting)
[ ] Defined success criteria and evaluation metrics
[ ] Selected base model appropriate for task and budget
[ ] Estimated compute requirements and costs
Data Preparation:
[ ] Collected 500+ high-quality examples (1000+ preferred)
[ ] Deduplicated and quality-filtered dataset
[ ] Split into train/evaluate/test (80/10/10)
[ ] Balanced task distribution
[ ] Verified formatting matches model's expected template
Training:
[ ] Selected method (LoRA for most cases, QLoRA for large models)
[ ] Configured hyperparameters (start with recommended defaults)
[ ] Enabled gradient checkpointing for memory efficiency
[ ] Set up evaluation during training (evaluate every N steps)
[ ] Monitored training loss curve for overfitting
Evaluation:
[ ] Evaluated on held-out test set
[ ] Compared against base model baseline
[ ] Checked for capability regression
[ ] Tested edge cases and adversarial inputs
[ ] Had domain expert review sample outputs
Deployment:
[ ] Merged LoRA weights (or deployed adapter separately)
[ ] Quantized for production (GPTQ, AWQ, or GGUF)
[ ] Set up serving infrastructure (vLLM, TGI, or Ollama)
[ ] Configured monitoring for output quality drift
[ ] Documented model card with training details
When to Use
Use this skill when:
- Designing or implementing llm fine tuner solutions
- Reviewing or improving existing llm fine tuner approaches
- Making architectural or implementation decisions about llm fine tuner
- Learning llm fine tuner patterns and best practices
- Troubleshooting llm fine tuner-related issues
Do NOT use this skill when:
- The question is about a fundamentally different technology domain
- A more specific sibling skill covers the exact topic needed
- The user needs a complete hands-on tutorial rather than expert guidance
Output Format
# Llm Fine Tuner Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]
Example
Input: "Help me implement llm fine tuner for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended llm fine tuner approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Edge Cases
- Legacy system integration: When llm fine tuner must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
- Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
- Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
- Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities