Mantis Interleaved Multi Image Instruction Tuning Arxiv 2405 01483v2

Mantis: Interleaved Multi-Image Instruction Tuning

One-line decision

Use this skill when you want to create instruction data for multi-image understanding tasks like comparison, reasoning across images, and temporal understanding. Avoid it when you only need single-image instruction data.

Skill metadata

• Skill type: multi-image-instruction-data
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Create multi-image instruction tuning data for training VLMs that can understand relationships, comparisons, and reasoning across multiple images in a single conversation.

Problem signature

• Modality: multiple images with instruction data requiring cross-image understanding.
• Data state: instruction data generated for multi-image tasks (comparison, temporal, spatial reasoning).
• Scale regime: 721K multi-image instruction samples.
• Model requirement: VLM with multi-image input support.

Use when

• You want a VLM that understands relationships between multiple images.
• You need training data for image comparison, temporal, or multi-view tasks.
• Your VLM supports multi-image input.

Do not use when

• Single-image understanding is sufficient.
• Your VLM cannot handle multiple images.
• You do not need cross-image reasoning.

Required inputs

• multi_image_sources: Sources of related image sets (temporal, comparison, multi-view).
• instruction_generator: Pipeline for generating multi-image instruction data.
• vlm_architecture: VLM supporting multiple image inputs.

Optional inputs

• image_relationship_labels: Labels describing relationships between images.

Outputs

• mantis_instruct_data: 721K multi-image instruction samples.
• mantis_model: VLM trained for multi-image understanding.

Assumptions and prerequisites

• Multi-image understanding is a distinct capability from single-image.
• Specialized instruction data is needed for cross-image reasoning.
• 721K samples provide sufficient multi-image training signal.

Procedure

1. Curate multi-image sources Action: Collect sets of related images from diverse sources. Why: Related images enable cross-image reasoning tasks. Note: See paper for details.
2. Generate multi-image instructions Action: Create instruction-response pairs requiring understanding across images. Why: Cross-image instructions teach multi-image reasoning. Note: See paper for details.
3. Cover diverse task types Action: Include comparison, temporal, spatial, and logical reasoning tasks. Why: Diverse tasks build comprehensive multi-image understanding. Note: See paper for details.
4. Train multi-image VLM Action: Fine-tune a VLM on the 721K multi-image instruction data. Why: Specialized training enables multi-image capabilities. Note: See paper for details.

Parameters to set

• image_pairs — Role: Number and type of image relationships. How to set: Include comparison, temporal, and multi-view pairs. Default/range: Diverse. Effect: More relationship types improve flexibility.
• instruction_types — Role: Types of multi-image instructions. How to set: Cover comparison, change detection, temporal, and reasoning. Default/range: Multiple types. Effect: Diverse types build comprehensive capability.

Validation checks

• The model should correctly reason across multiple images.
• Multi-image benchmarks should show improvement.
• Single-image performance should not degrade.

Failure modes

• The model may not distinguish relevant from irrelevant images.
• Multi-image data generation is more complex than single-image.
• Interleaved image-text formatting may be inconsistent.

Adaptation notes for VLM training

• Multi-image instruction data is essential for document comparison, temporal understanding, etc.
• Combine with single-image data for balanced training.
• The multi-image approach extends to video understanding.

Implementation notes

• Use consistent image ordering conventions.
• Track per-task multi-image performance.
• Handle variable numbers of images per instruction.

Evidence from the paper

• Mantis creates 721K multi-image instruction samples for cross-image understanding.
• The model handles comparison, temporal, and spatial reasoning across images.
• Multi-image instruction tuning is a distinct and important capability.
• Mantis achieves state-of-the-art on multi-image benchmarks.

Source paper

• Title: Mantis: Interleaved Multi-Image Instruction Tuning
• Year: 2024
• Venue: arXiv
• Paper ID: arxiv-2405.01483v2
• URL: http://arxiv.org/abs/2405.01483v2
• arXiv ID: 2405.01483v2