LLM Fine-Tuning
LoRA/QLoRA (Parameter-Efficient)
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],
lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, config)
Data Formats
- Alpaca:
{"instruction": ..., "input": ..., "output": ...}
- ShareGPT:
{"conversations": [{"from": "human", "value": ...}, ...]}
- OpenAI:
{"messages": [{"role": "user", "content": ...}, ...]}
Alignment
- DPO: direct preference optimization, no reward model needed
- ORPO: odds ratio preference, single-stage
- SimPO: simple preference optimization with reference-free margin
Memory Optimization
- Gradient checkpointing:
model.gradient_checkpointing_enable()
- DeepSpeed ZeRO Stage 2/3 for multi-GPU
- QLoRA: 4-bit base model + LoRA adapters
Evaluation
Perplexity, downstream benchmarks, MT-Bench/AlpacaEval, human eval
Key Libraries
transformers, trl, peft, unsloth, axolotl
1---2name: llm-finetuning3description: Fine-tune LLMs with LoRA/QLoRA, instruction tuning, and alignment (DPO/ORPO).4---56# LLM Fine-Tuning78## LoRA/QLoRA (Parameter-Efficient)9```python10from peft import LoraConfig, get_peft_model11config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],12 lora_dropout=0.05, task_type="CAUSAL_LM")13model = get_peft_model(model, config)14```1516## Data Formats17- **Alpaca**: `{"instruction": ..., "input": ..., "output": ...}`18- **ShareGPT**: `{"conversations": [{"from": "human", "value": ...}, ...]}`19- **OpenAI**: `{"messages": [{"role": "user", "content": ...}, ...]}`2021## Alignment22- **DPO**: direct preference optimization, no reward model needed23- **ORPO**: odds ratio preference, single-stage24- **SimPO**: simple preference optimization with reference-free margin2526## Memory Optimization27- Gradient checkpointing: `model.gradient_checkpointing_enable()`28- DeepSpeed ZeRO Stage 2/3 for multi-GPU29- QLoRA: 4-bit base model + LoRA adapters3031## Evaluation32Perplexity, downstream benchmarks, MT-Bench/AlpacaEval, human eval3334## Key Libraries35transformers, trl, peft, unsloth, axolotl
aselimc