Jiachen-T-Wang Flamingo: A Visual Language Model for Few-Shot Learning
One-line decision
Use this skill when you need to collect and train on interleaved image-text web data for few-shot visual learning. Avoid it when you only have paired image-caption data without interleaved document structure.
Skill metadata
- Skill type: interleaved-data-collection
- Paper kind: operational-method
- Actionability: high
- Evidence quality: full_paper
Goal
Collect interleaved image-text web documents and combine with paired image-text data to train a visual language model capable of rapid few-shot adaptation to visual tasks.
Problem signature
- Modality: interleaved image-text sequences from web pages plus standard image-text pairs.
- Data state: web pages with naturally interleaved images and text, plus curated image-text pair datasets.
- Scale regime: 185M image-text pairs + 43M web pages with interleaved images and text.
- Model requirement: Perceiver-based vision encoder cross-attending to a frozen Chinchilla LLM.
Use when
- You want a VLM capable of few-shot learning from interleaved image-text examples.
- You can collect web pages with naturally interleaved images and text.
- You need in-context visual learning without fine-tuning.
Do not use when
- You only have isolated image-caption pairs without document structure.
- You need a model for single-image understanding only.
- You cannot afford training on both interleaved and paired data.
Required inputs
- interleaved_web_documents: Web pages containing naturally interleaved images and text (M3W dataset: 43M pages).
- paired_image_text: Standard image-caption pair datasets (LAION-2B, etc.).
- frozen_llm: Pre-trained language model to serve as the base (Chinchilla 70B).
Optional inputs
- video_text_pairs: Video-text pairs for temporal understanding.
Outputs
- flamingo_model: VLM capable of few-shot visual reasoning from interleaved examples.
- m3w_dataset: 43M interleaved web documents with images.
Assumptions and prerequisites
- Interleaved image-text web data teaches in-context multimodal reasoning.
- Combining interleaved and paired data produces complementary capabilities.
- Few-shot learning emerges from training on naturally interleaved sequences.
Procedure
- Collect M3W dataset Action: Extract web pages with interleaved images and text from Common Crawl, keeping document structure. Why: Interleaved data enables few-shot in-context learning. Note: Keep images in their original position within the text.
- Curate paired image-text data Action: Combine LAION-2B and other paired datasets filtered for quality. Why: Paired data provides clean image-text alignment. Note: See paper for details.
- Mix interleaved and paired data Action: Alternate between interleaved and paired batches during training. Why: Both data types contribute complementary capabilities. Note: See paper for details.
- Train with frozen LLM Action: Train perceiver resampler and cross-attention layers while keeping the LLM frozen. Why: Preserves the LLM's language capabilities while adding vision. Note: See paper for details.
- Evaluate few-shot performance Action: Test few-shot visual QA, captioning, and classification with in-context examples. Why: Few-shot ability is the key capability enabled by interleaved training. Note: See paper for details.
Parameters to set
- interleaved_ratio — Role: Fraction of training from interleaved vs paired data. How to set: Mix interleaved and paired data in training batches. Default/range: Not specified exactly. Effect: More interleaved data improves few-shot; more paired improves zero-shot.
- max_images_per_sequence — Role: Maximum images in one interleaved sequence. How to set: 5-10 images per sequence. Default/range: 5. Effect: More images per sequence enable longer few-shot contexts.
- perceiver_queries — Role: Number of queries in perceiver resampler. How to set: 64 queries per image. Default/range: 64. Effect: Fewer queries compress visual information more aggressively.
Validation checks
- Few-shot VQA accuracy should increase with more in-context examples.
- The model should outperform fine-tuned baselines in low-data regimes.
- Zero-shot performance should be competitive with CLIP-based models.
Failure modes
- Interleaved web data may contain noisy or irrelevant image-text associations.
- The perceiver resampler may discard fine-grained visual details.
- Few-shot performance is sensitive to the format and ordering of in-context examples.
Adaptation notes for VLM training
- The interleaved data format has been adopted by many subsequent VLMs (Emu, IDEFICS, etc.).
- Collect domain-specific interleaved documents for specialized few-shot capabilities.
- The perceiver resampler pattern is reusable across VLM architectures.
Implementation notes
- Preserve document structure (image positions) during web crawling.
- Use cross-attention rather than concatenation for efficiency with multiple images.
- Pre-compute vision features and cache them for faster training iterations.
Evidence from the paper
- Flamingo achieves state-of-the-art few-shot performance on 16 multimodal benchmarks.
- Interleaved image-text training (M3W) is critical for few-shot in-context learning.
- The model handles arbitrary numbers of interleaved images and text.
- Flamingo 80B sets new state-of-the-art with just 4 few-shot examples on many tasks.
Source paper
- Title: Flamingo: A Visual Language Model for Few-Shot Learning
- Year: 2022
- Venue: NeurIPS
- Paper ID: arxiv-2204.14198v2
- URL: http://arxiv.org/abs/2204.14198v2
- arXiv ID: 2204.14198v2