# Lora Finetune

> LoRA fine-tuning pipeline for Stable Diffusion on Apple Silicon — dataset prep, training, evaluation with LLM-as-judge scoring. Use when fine-tuning image generation models for consistent style, custom characters, or domain-specific visuals. Requires Python with torch and diffusers.

- Skill: `modbender/lora-finetune` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add modbender/lora-finetune`
- Raw SKILL.md: https://api.skillmd.com/api/skills/modbender/lora-finetune/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: modbender (https://skillmd.com/u/modbender)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/modbender/lora-finetune

---


# LoRA Fine-Tuning (Apple Silicon)

Train custom LoRA adapters for Stable Diffusion 1.5 on Mac hardware. Tested on M4 24GB — produces 3.1MB weight files in ~15 minutes at 500 steps.

## Hardware Requirements

| Config | Model | Resolution | VRAM |
|---|---|---|---|
| M4 24GB | SD 1.5 | 512×512 | ✅ Works |
| M4 24GB | SDXL | 512×512 | ⚠️ Tight, may OOM |
| M4 24GB | FLUX.1-schnell | Any | ❌ OOMs |
| M4 Pro 48GB | SDXL | 1024×1024 | ✅ Estimated |

## Training Pipeline

1. **Prepare dataset:** 15-25 images in consistent style, 512×512, with text captions
2. **Train LoRA:** 500 steps, learning rate 1e-4, rank 4
3. **Evaluate:** Generate test images, compare base vs LoRA vs reference (Gemini/DALL-E)
4. **Score:** LLM-as-judge rates each on style consistency, quality, prompt adherence

## Quick Start

```bash
# Prepare training images in a folder
ls training_data/
# image_001.png  image_001.txt  image_002.png  image_002.txt ...

# Train (see scripts/train_lora.py for full options)
python3 scripts/train_lora.py \
  --data_dir ./training_data \
  --output_dir ./lora_weights \
  --steps 500 \
  --lr 1e-4 \
  --rank 4
```

## Evaluation with LLM-as-Judge

```python
# Compare base model vs LoRA vs commercial (Gemini/DALL-E)
# Pixtral Large scores each image 1-10 on:
# - Style consistency with training data
# - Image quality and coherence
# - Prompt adherence

# Our results: Base 6.8 → LoRA 9.0 → Gemini 9.5
# Lesson: Gemini wins without training, but LoRA closes the gap significantly
```

## Key Lessons

- **float32 required on MPS** — float16 silently produces NaN on Apple Silicon for SD pipelines
- **mflux is faster than PyTorch MPS for FLUX** (~105s vs ~90min) but doesn't support LoRA training
- **SD 1.5 is the ceiling for 24GB** — FLUX LoRA OOMs even with gradient checkpointing
- **15-25 images is the sweet spot** — fewer undertrain, more doesn't help proportionally
- **Gemini (Imagen 4.0) beats fine-tuned SD 1.5** with zero training — use commercial APIs for production, LoRA for experimentation and offline use

## Files

- `scripts/train_lora.py` — Training script with Apple Silicon MPS support
- `scripts/compare_models.py` — LLM-as-judge evaluation comparing base vs LoRA vs reference