adu2021 Title: Interleave Text and Visual Reasoning for Adaptive Problem Solving
Multimodal reasoning traditionally treats text and images as separate streams or forces them into a single unified representation. ThinkMorph demonstrates that treating text and images as complementary modalities that jointly advance reasoning toward solutions yields emergent capabilities: models spontaneously switch between text-only and visual reasoning based on task requirements, and interleaved reasoning shows better scaling curves under best-of-N sampling than text-only approaches.
The core insight is that visual reasoning excels at spatial manipulation tasks while textual reasoning maintains logical coherence. By interleaving token generation across both modalities, you create a unified problem-solving process where each modality amplifies the other's strengths.
Core Concept
Interleaved Multimodal Reasoning structures a chain-of-thought as alternating text and image thoughts, where:
- Text tokens maintain logical flow and coherence
- Image tokens represent visual manipulations (edits, annotations, highlights)
- The model learns when to generate each modality based on task requirements
- Training on only ~24K examples enables generalization to out-of-domain tasks
This approach differs from sequential multimodal reasoning (generate all text, then all images) or merged representations by creating tight coupling: each reasoning step may involve both modalities simultaneously.
Architecture Overview
- Base Model: Unified architecture (e.g., Bagel-7B) with shared token vocabulary for text and image patches
- Reasoning Modes: Three training modes: text-only, visual-only, interleaved
- Loss Functions: Dual objectives combining MSE for image reconstruction and NLL for text prediction
- Dataset: ~24K curated traces across four vision-centric task categories with human quality control
- Evaluation: Vision-centric benchmarks (SAT, MMVP, VStar, BLINK, CV-Bench) plus out-of-domain generalization tests
Implementation Steps
1. Curate High-Quality Interleaved Reasoning Traces
Create training data by collecting problem-solving sequences where humans solve vision tasks while generating both text thoughts (reasoning steps) and visual annotations (highlights, edits, overlays). Use GPT-4 to generate candidate traces, then human raters filter for coherence and correctness.
# Example structure of an interleaved trace
trace = {
"task": "locate_object_in_image",
"steps": [
{"type": "text", "content": "I need to scan the image systematically"},
{"type": "image", "content": "<visual_annotation>", "action": "highlight_region"},
{"type": "text", "content": "The object appears in the upper-left area"},
{"type": "image", "content": "<zoomed_patch>", "action": "zoom_in"},
]
}
# Organize by task category: jigsaw (6K), navigation (6K), search (7K), chart (6K)
2. Tokenize Mixed Modality Sequences
Represent images as sequences of tokens from an image vocabulary (e.g., using a VQ-VAE codebook). Interleave text and image token sequences into a single stream, enabling standard transformer training. Use special tokens to mark modality boundaries.
# Tokenization of interleaved sequence
text_tokens = tokenizer.encode("I need to locate") # [512, 1023, 450, ...]
image_tokens = vqvae.encode(image_patch) # [2048, 2051, 2050, ...]
# Interleave: mark boundaries with modality tokens
interleaved = [512, 1023, 450, MODALITY_IMG, 2048, 2051, MODALITY_TXT, ...]
# Standard next-token prediction loss applies uniformly
3. Implement Dual Loss Training
Combine reconstruction loss for images (MSE on patch space) with NLL for text tokens. Weight appropriately based on modality mix in batches.
# Dual loss computation
def compute_loss(predictions, targets, modalities):
text_mask = modalities == MODALITY_TXT
image_mask = modalities == MODALITY_IMG
text_loss = F.cross_entropy(predictions[text_mask], targets[text_mask])
image_loss = F.mse_loss(predictions[image_mask], targets[image_mask])
return text_loss + lambda_image * image_loss
4. Train on Diverse Vision Tasks
Fine-tune the base model on the curated dataset using causal language modeling. The model learns to map task descriptions to appropriate reasoning patterns through exposure to diverse problem types.
# Training loop structure
for epoch in range(num_epochs):
for batch in dataloader:
traces = batch["interleaved_tokens"] # Mix of text and image tokens
loss = compute_loss(model(traces), traces)
loss.backward()
optimizer.step()
Practical Guidance
When to Use:
- Vision-centric reasoning requiring spatial manipulation (puzzles, navigation, visual search)
- Tasks where text reasoning alone leaves important spatial information implicit
- Scenarios where test-time compute budgeting is critical (interleaved-N=4 > text-only-N=8)
Hyperparameters:
- Data curation: Aim for 4-6K high-quality examples per task category
- λ_image: Weight for image reconstruction loss (typically 0.5-1.0)
- Training epochs: 2-3 passes usually sufficient with curriculum (easier tasks first)
When NOT to Use:
- Pure text reasoning tasks (language understanding, knowledge QA)
- Tasks without clear visual manipulation requirements
- Settings where latency is critical (image tokenization adds overhead)
Pitfalls:
- Insufficient data curation: Poor-quality traces teach spurious correlations between text and images
- Modality imbalance: Too few image tokens causes visual reasoning to remain underdeveloped
- Overfitting to task distribution: The 24K examples must span diverse problem types or generalization fails
Emergent Properties to Exploit:
- Autonomous mode-switching: Despite training only on interleaved data, models spontaneously use text-only reasoning when appropriate (~5% of cases), achieving higher accuracy than forced interleaving
- Superior scaling under best-of-N: Interleaved reasoning explores more diverse trajectories, yielding +5-8% gains over text-only under sampling
- Novel visual manipulations: Models generate unseen edits during inference (perspective transforms, inpainting) without explicit training
Integration Point: Combine with VLM fine-tuning pipelines—interleaved reasoning complements supervised fine-tuning on domain-specific tasks.