Improved Baselines With Visual Instruction Tuning Arxiv 2310 03744v2

Improved Baselines with Visual Instruction Tuning

One-line decision

Use this skill when you want to curate a high-quality academic-task-oriented instruction dataset for VLM fine-tuning. Avoid it when you need only synthetic conversation data or cannot access the academic VQA datasets.

Skill metadata

• Skill type: instruction-data-curation
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Curate a high-quality, diverse instruction-tuning dataset by combining academic task-oriented VQA data with GPT-4V-generated visual conversations to improve VLM fine-tuning baselines.

Problem signature

• Modality: image-text instruction-response pairs mixing academic VQA and synthetic conversation data.
• Data state: combination of existing academic VQA datasets reformatted as instructions plus synthetic conversation data.
• Scale regime: 665K instruction samples for fine-tuning.
• Model requirement: CLIP ViT-L/14@336 + Vicuna/LLaMA with two-layer MLP projection.

Use when

• You want to improve VLM instruction-following by incorporating academic VQA datasets.
• You need a curated mix of data types (VQA, OCR, region-level, conversation) for fine-tuning.
• You want simple but effective improvements over the original LLaVA recipe.

Do not use when

• You cannot access the academic datasets (VQAv2, GQA, OCR-VQA, etc.).
• You only need a model for a single specific task.
• You need multi-image or video understanding.

Required inputs

• academic_vqa_datasets: Reformatted VQAv2, GQA, OCR-VQA, TextVQA, and VisualGenome QA as instruction-following format.
• synthetic_conversation_data: GPT-4V-generated visual conversation data (LLaVA-Instruct-150K).
• pretrained_components: CLIP ViT-L/14@336 and Vicuna/LLaMA language model.

Optional inputs

• sharegpt_data: Additional ShareGPT text-only conversation data mixed in for language ability preservation.

Outputs

• curated_instruction_mix: 665K instruction-following samples balancing academic and synthetic data.
• llava_1_5_model: Improved VLM baseline (LLaVA-1.5) achieving strong benchmark performance.

Assumptions and prerequisites

• Academic VQA datasets provide grounded, factual supervision that reduces hallucination.
• A balanced mix of data types produces a more capable generalist model.
• Simple architectural improvements (MLP projection, higher resolution) compound with better data.

Procedure

1. Reformat academic datasets as instructions Action: Convert VQAv2, GQA, OCR-VQA, TextVQA, and VisualGenome region QA into instruction-response format. Why: Standardizes diverse datasets into a unified training format. Note: See paper for details.
2. Combine with synthetic conversation data Action: Mix reformatted academic data with LLaVA-Instruct-150K conversation data. Why: Provides both factual grounding and conversational ability. Note: See paper for details.
3. Add text-only conversation data Action: Include ShareGPT text-only data to preserve language model capabilities. Why: Prevents catastrophic forgetting of language abilities during visual fine-tuning. Note: See paper for details.
4. Pre-train projection with CC595K Action: Train 2-layer MLP projection on a filtered 595K subset of CC3M. Why: Aligns visual features to language model space. Note: Higher resolution (336px) and MLP projection are key improvements.
5. Fine-tune on curated instruction mix Action: Fine-tune the full model on the 665K curated instruction dataset. Why: Teaches the model to follow diverse visual instructions. Note: See paper for details.

Parameters to set

• image_resolution — Role: Input image resolution for CLIP encoder. How to set: Use 336x336 for improved detail recognition. Default/range: 336. Effect: Higher resolution significantly improves OCR and fine-grained tasks.
• projection_type — Role: Vision-language bridge architecture. How to set: Use 2-layer MLP instead of linear. Default/range: 2-layer MLP. Effect: MLP projection outperforms linear projection.
• data_mix — Role: Ratio of academic vs synthetic data. How to set: Include all academic data plus conversation data. Default/range: ~515K academic + 150K synthetic. Effect: Academic data grounds the model; synthetic data teaches conversation.

Validation checks

• Performance should improve on VQAv2, GQA, TextVQA, and conversational benchmarks.
• Hallucination rate should decrease with academic data inclusion.
• The model should maintain strong language capabilities from ShareGPT mixing.

Failure modes

• Over-reliance on academic data may reduce conversational naturalness.
• Dataset-specific biases from VQA benchmarks may transfer to the model.
• The mix ratio needs tuning; wrong balance can hurt specific capabilities.

Adaptation notes for VLM training

• The LLaVA-1.5 data recipe has become a standard baseline for VLM instruction tuning.
• Extend with additional academic datasets (DocVQA, ChartQA) for document understanding.
• The MLP projection + academic data recipe transfers to other VLM architectures.

Implementation notes

• Use the LLaVA codebase for data formatting and training pipeline.
• Academic datasets need careful reformatting to preserve answer quality.
• Monitor per-dataset performance during training to detect imbalances.

Evidence from the paper

• LLaVA-1.5 achieves state-of-the-art results across 11 benchmarks with a simple recipe: MLP projection, academic task data, and higher resolution.
• Adding academic VQA data (VQAv2, GQA, OCR-VQA, TextVQA) to instruction tuning significantly improves grounded understanding.
• The 665K instruction mix balances factual accuracy from academic data with conversational ability from synthetic data.
• LLaVA-1.5 13B achieves 80.0% on VQAv2, 63.3% on GQA, and 61.3% on TextVQA.

Source paper

• Title: Improved Baselines with Visual Instruction Tuning
• Year: 2023
• Venue: CVPR
• Paper ID: arxiv-2310.03744v2
• URL: http://arxiv.org/abs/2310.03744v2
• arXiv ID: 2310.03744v2