Visual Instruction Tuning Arxiv 2304 08485v2

Visual Instruction Tuning

One-line decision

Use this skill when you want to generate multimodal instruction-following data by prompting a text-only LLM with image captions and bounding boxes. Avoid it when you have abundant human-annotated instruction data or need the LLM to see actual images during data generation.

Skill metadata

• Skill type: instruction-data-synthesis
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Generate multimodal instruction-following data by leveraging text-only GPT-4 with image captions and bounding boxes as context, then use this synthetic data to train a vision-language model (LLaVA) for visual instruction following.

Problem signature

• Modality: image-text instruction-following pairs synthesized from image captions and bounding box annotations.
• Data state: COCO images with existing captions and bounding box annotations used to generate instruction data via LLM.
• Scale regime: 158K instruction-following samples synthesized from COCO images.
• Model requirement: A text-only LLM (GPT-4) for data generation; CLIP ViT-L + Vicuna for the trained VLM.

Use when

• You need instruction-following data for visual tasks but lack human annotations.
• You have image captions and object annotations that can seed LLM-based data generation.
• You want to fine-tune a VLM for conversational visual understanding.

Do not use when

• You have abundant human-annotated visual instruction data.
• The images lack any existing annotations (captions, boxes) to ground the LLM generation.
• You need pixel-perfect or region-level instruction data.

Required inputs

• source_images: Images with existing annotations (e.g., COCO images with captions and bounding boxes).
• text_llm: A powerful text-only LLM (e.g., GPT-4) to generate instruction-response pairs from textual descriptions of images.
• prompt_templates: Prompt templates for conversation, detail description, and complex reasoning data types.

Optional inputs

• human_seed_examples: Small set of human-written examples to guide the LLM in each data type.
• quality_filter: Rules to filter out low-quality or hallucinated instruction-response pairs.

Outputs

• instruction_following_data: 158K multimodal instruction-following samples covering conversation (58K), detail description (23K), and complex reasoning (77K).
• trained_vlm: LLaVA model fine-tuned on the synthetic instruction data.

Assumptions and prerequisites

• A text-only LLM can generate faithful visual instruction data from textual descriptions of images.
• Image captions and bounding boxes provide sufficient context for the LLM to reason about the image.
• Three types of instruction data (conversation, detail, reasoning) provide comprehensive coverage.

Procedure

1. Prepare image context as text Action: Convert image annotations (captions, bounding boxes with labels) into textual descriptions for the LLM. Why: Text-only LLMs cannot process images directly. Note: Use COCO captions and detection annotations.
2. Design prompt templates for three data types Action: Create few-shot prompts for conversation, detailed description, and complex reasoning. Why: Different data types teach different visual understanding capabilities. Note: See paper for details.
3. Generate instruction data with LLM Action: Prompt GPT-4 with image context and templates to generate question-answer pairs. Why: LLM generates diverse, high-quality instruction-response pairs. Note: Generate 158K samples total across three types.
4. Pre-train visual projection Action: Train a linear projection from CLIP ViT-L to Vicuna using CC3M caption data. Why: Aligns visual features with the language model's input space. Note: See paper for details.
5. Fine-tune on instruction data Action: Fine-tune the full LLaVA model on the 158K synthetic instruction samples. Why: Teaches the model to follow visual instructions in conversation. Note: See paper for details.

Parameters to set

• data_type_ratio — Role: Balance between conversation, detail, and reasoning samples. How to set: Use roughly 37% conversation, 14% detail, 49% reasoning. Default/range: 58K:23K:77K. Effect: Reasoning-heavy mix improves complex visual understanding.
• projection_type — Role: Architecture bridging vision encoder to LLM. How to set: Start with linear projection; MLP may improve later. Default/range: Linear. Effect: Linear is simple and effective for initial alignment.
• base_llm — Role: Language model backbone. How to set: Use Vicuna-13B for best quality. Default/range: Vicuna-7B or 13B. Effect: Larger LLMs generate better responses.

Validation checks

• Generated instruction data should be visually grounded (not hallucinated from text alone).
• LLaVA should improve on visual conversation benchmarks after fine-tuning.
• The three data types should be distinguishable in style and complexity.

Failure modes

• The text-only LLM may hallucinate visual details not present in the captions or boxes.
• Generated data inherits biases from COCO annotations and the LLM.
• Complex reasoning data may contain logical errors when the LLM lacks visual grounding.

Adaptation notes for VLM training

• This data generation pipeline has been widely adopted and extended (LLaVA-1.5, LLaVA-NeXT, etc.).
• Replace GPT-4 with Claude or open-source LLMs for cost-effective generation.
• Extend the pipeline to video, 3D, or document understanding by changing the context format.

Implementation notes

• Use structured prompts with clear role separation (system, human, assistant).
• Generate data in batches and inspect random samples for quality.
• Store raw LLM responses alongside parsed instruction pairs for debugging.

Evidence from the paper

• LLaVA generates 158K language-image instruction-following samples using GPT-4 with image captions and bounding boxes as context.
• The generated data covers three types: multi-turn conversation (58K), detailed description (23K), and complex reasoning (77K).
• LLaVA, trained on this synthetic data, achieves an 85.1% relative score compared to GPT-4 on a synthetic multimodal benchmark.
• Visual instruction tuning with machine-generated data is an effective and affordable approach to building general-purpose visual assistants.

Source paper

• Title: Visual Instruction Tuning
• Year: 2023
• Venue: NeurIPS
• Paper ID: arxiv-2304.08485v2
• URL: http://arxiv.org/abs/2304.08485v2
• arXiv ID: 2304.08485v2