Overview
Unsloth provides 2x faster QLoRA training with 50% less memory via optimized kernels. Supports Llama, Mistral, Gemma, Qwen 2.5, DeepSeek, Phi, Yi, and Falcon with Flash Attention.
Installation
uv pip install unsloth
QLoRA Fine-Tuning
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
max_seq_length=4096,
dtype=torch.bfloat16,
load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
model, r=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_alpha=16, use_gradient_checkpointing="unsloth",
)
print(model.print_trainable_parameters())
Inference
FastLanguageModel.for_inference(model)
inputs = tokenizer(["Describe quantum computing."], return_tensors="pt").to("cuda")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=256)[0]))
References
1---2name: unsloth3description: Fast QLoRA/QLoRA fine-tuning with 2x faster training and 50% less memory. Supports Llama, Mistral, Gemma, Qwen, DeepSeek, Phi, Yi, Falcon. Flash Attention, 4-bit quantization. No quality loss.4---5## Overview67Unsloth provides 2x faster QLoRA training with 50% less memory via optimized kernels. Supports Llama, Mistral, Gemma, Qwen 2.5, DeepSeek, Phi, Yi, and Falcon with Flash Attention.89## Installation1011```bash12uv pip install unsloth13```1415## QLoRA Fine-Tuning1617```python18from unsloth import FastLanguageModel19import torch2021model, tokenizer = FastLanguageModel.from_pretrained(22 model_name="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",23 max_seq_length=4096,24 dtype=torch.bfloat16,25 load_in_4bit=True,26)27model = FastLanguageModel.get_peft_model(28 model, r=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],29 lora_alpha=16, use_gradient_checkpointing="unsloth",30)31print(model.print_trainable_parameters())32```3334## Inference3536```python37FastLanguageModel.for_inference(model)38inputs = tokenizer(["Describe quantum computing."], return_tensors="pt").to("cuda")39print(tokenizer.decode(model.generate(**inputs, max_new_tokens=256)[0]))40```4142## References43- [Unsloth GitHub](https://github.com/unslothai/unsloth)44- [Unsloth docs](https://docs.unsloth.ai/)
mkurman