Visual Prompt Tuning Arxiv 2203 12119v2

Visual Prompt Tuning

One-line decision

Use this skill when you want to adapt a pre-trained vision model to a new task by prepending learnable visual prompt tokens, training only the prompts. Avoid it when full fine-tuning is affordable or LoRA is preferred.

Skill metadata

• Skill type: parameter-efficient-visual-adaptation
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Adapt pre-trained vision models to new tasks by prepending a small number of learnable visual prompt tokens to the input, training only the prompt tokens while keeping the model frozen.

Problem signature

• Modality: images with learnable visual prompt tokens.
• Data state: task-specific data used to train visual prompts only.
• Scale regime: any dataset; only prompt tokens are trained.
• Model requirement: Pre-trained ViT with frozen weights.

Use when

• You want extremely parameter-efficient visual adaptation.
• You need to adapt to many tasks without full fine-tuning.
• You want to keep the vision model frozen.

Do not use when

• Full fine-tuning is affordable.
• LoRA provides better results.
• You need to change the model architecture.

Required inputs

• pretrained_vit: Frozen pre-trained ViT.
• task_data: Task-specific training data.
• visual_prompts: Learnable prompt tokens to prepend.

Optional inputs

• deep_prompting: Prompts at multiple layers (VPT-Deep).

Outputs

• adapted_model: Vision model adapted through visual prompts.
• prompt_tokens: Learned task-specific visual prompt tokens.

Assumptions and prerequisites

• A small number of prompt tokens can adapt a frozen model.
• Visual prompting is analogous to language prompting.
• Frozen models preserve general representations.

Procedure

1. Initialize visual prompts Action: Create learnable prompt tokens. Why: Prompts are the only trainable parameters. Note: See paper for details.
2. Prepend to input Action: Add prompts before image patch tokens. Why: Prompts influence model processing. Note: See paper for details.
3. Train prompts only Action: Train only the prompt tokens on task data. Why: Extremely parameter-efficient adaptation. Note: See paper for details.
4. Evaluate adaptation Action: Test on the target task. Why: Validates prompt-based adaptation. Note: See paper for details.

Parameters to set

• num_prompt_tokens — Role: Number of visual prompt tokens. How to set: 10-100 tokens. Default/range: 50. Effect: More tokens provide more adaptation capacity.
• prompt_depth — Role: Where to add prompts. How to set: VPT-Shallow (input only) or VPT-Deep (all layers). Default/range: VPT-Deep. Effect: Deep prompting provides more capacity.

Validation checks

• Prompt-tuned model should approach full fine-tuning quality.
• Only prompt tokens should be updated.
• The approach should work across diverse tasks.

Failure modes

• Very few prompts may be insufficient.
• Some tasks may require more adaptation.
• Prompt-based adaptation may not match LoRA quality.

Adaptation notes for VLM training

• Visual prompt tuning provides efficient VLM adaptation.
• Apply to domain-specific VLM adaptation with minimal data.
• Compare to LoRA for the best efficient adaptation method.

Implementation notes

• Implement prompt tokens as learnable parameters.
• Use VPT-Deep for best results.
• Compare to full fine-tuning and LoRA.

Evidence from the paper

• Visual Prompt Tuning adapts ViTs with only 0.1% trainable parameters.
• VPT matches full fine-tuning on most tasks.
• Visual prompts are analogous to language prompts for adaptation.
• The approach enables efficient multi-task VLM adaptation.

Source paper

• Title: Visual Prompt Tuning
• Year: 2022
• Venue: ECCV
• Paper ID: arxiv-2203.12119v2
• URL: http://arxiv.org/abs/2203.12119v2
• arXiv ID: 2203.12119v2