Lora Low Rank Adaptation Of Large Language Models Arxiv 2106 09685v2

LoRA: Low-Rank Adaptation of Large Language Models

One-line decision

Use this skill when you want to efficiently fine-tune LLMs/VLMs on domain-specific data using low-rank adapter matrices instead of full fine-tuning. Avoid it when you can afford full fine-tuning or need maximum model capacity.

Skill metadata

• Skill type: parameter-efficient-fine-tuning
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Enable efficient fine-tuning of large language models by adding low-rank adapter matrices to frozen pre-trained weights, dramatically reducing the number of trainable parameters while maintaining performance.

Problem signature

• Modality: any modality; widely used for VLM fine-tuning.
• Data state: domain-specific or task-specific data for efficient fine-tuning.
• Scale regime: any fine-tuning dataset size.
• Model requirement: Any transformer-based model (LLM, VLM).

Use when

• You want to fine-tune on domain-specific data with limited compute.
• You need parameter-efficient adaptation.
• You want to maintain multiple task-specific adapters.

Do not use when

• Full fine-tuning is feasible and preferred.
• You need maximum model capacity for your task.
• Your model is already small enough for full fine-tuning.

Required inputs

• pretrained_model: Pre-trained model to adapt.
• fine_tuning_data: Task-specific or domain-specific data.
• lora_config: Rank, target modules, and alpha configuration.

Optional inputs

• merge_weights: Option to merge LoRA weights into the base model.

Outputs

• lora_adapters: Small adapter weight files for each task.
• adapted_model: Model adapted for the specific task.

Assumptions and prerequisites

• Weight updates during fine-tuning have low intrinsic rank.
• Low-rank adapters capture sufficient task-specific information.
• LoRA matches full fine-tuning on most tasks.

Procedure

1. Configure LoRA parameters Action: Set rank, target modules (attention layers), and alpha. Why: Configuration determines adaptation capacity and efficiency. Note: See paper for details.
2. Freeze base model Action: Freeze all pre-trained parameters. Why: Only adapter parameters are trained. Note: See paper for details.
3. Train LoRA adapters Action: Fine-tune only the low-rank adapter matrices on task data. Why: Parameter-efficient training on limited data. Note: See paper for details.
4. Merge or switch adapters Action: Optionally merge adapters into base weights or switch between task adapters. Why: Merged weights have zero inference overhead. Note: See paper for details.

Parameters to set

• rank — Role: Rank of the low-rank adaptation. How to set: 8-64 for most tasks. Default/range: 16. Effect: Higher rank increases capacity but also parameters.
• alpha — Role: Scaling factor for LoRA updates. How to set: Often set equal to rank or 2x rank. Default/range: 16-32. Effect: Higher alpha increases the magnitude of adaptation.
• target_modules — Role: Which layers get LoRA adapters. How to set: Attention layers (Q, K, V, O projections). Default/range: Attention layers. Effect: More modules increase capacity.

Validation checks

• LoRA should match or approach full fine-tuning performance.
• Training should be significantly faster than full fine-tuning.
• Adapter sizes should be small (MBs, not GBs).

Failure modes

• Very low rank may be insufficient for complex tasks.
• Some tasks may genuinely require full fine-tuning.
• LoRA may not work well for architectural modifications.

Adaptation notes for VLM training

• LoRA is the standard method for VLM fine-tuning on domain-specific data.
• Use LoRA for efficient VLM adaptation to medical, scientific, or other domains.
• Combine with curated domain-specific data for targeted VLM improvement.

Implementation notes

• Use the PEFT library for LoRA implementation.
• Start with rank 16 and adjust based on performance.
• Merge weights for deployment to avoid inference overhead.

Evidence from the paper

• LoRA matches full fine-tuning on most NLP tasks with <0.1% of trainable parameters.
• Low-rank adapters effectively capture task-specific knowledge.
• LoRA has become the standard for parameter-efficient VLM fine-tuning.
• The approach enables maintaining multiple task-specific adapters efficiently.

Source paper

• Title: LoRA: Low-Rank Adaptation of Large Language Models
• Year: 2022
• Venue: ICLR
• Paper ID: arxiv-2106.09685v2
• URL: http://arxiv.org/abs/2106.09685v2
• arXiv ID: 2106.09685v2