# Thinkmorph Interleaved Multimodal Reasoning

> Train unified models to generate interleaved reasoning steps combining text and image thoughts as complementary modalities, enabling adaptive behavior like autonomous mode-switching and superior test-time scaling for vision-centric tasks without requiring external guidance.

- Skill: `adu2021/thinkmorph-interleaved-multimodal-reasoning` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/thinkmorph-interleaved-multimodal-reasoning`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/thinkmorph-interleaved-multimodal-reasoning/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/thinkmorph-interleaved-multimodal-reasoning

---


# Title: Interleave Text and Visual Reasoning for Adaptive Problem Solving

Multimodal reasoning traditionally treats text and images as separate streams or forces them into a single unified representation. ThinkMorph demonstrates that treating text and images as **complementary modalities** that jointly advance reasoning toward solutions yields emergent capabilities: models spontaneously switch between text-only and visual reasoning based on task requirements, and interleaved reasoning shows better scaling curves under best-of-N sampling than text-only approaches.

The core insight is that visual reasoning excels at spatial manipulation tasks while textual reasoning maintains logical coherence. By interleaving token generation across both modalities, you create a unified problem-solving process where each modality amplifies the other's strengths.

## Core Concept

**Interleaved Multimodal Reasoning** structures a chain-of-thought as alternating text and image thoughts, where:
- Text tokens maintain logical flow and coherence
- Image tokens represent visual manipulations (edits, annotations, highlights)
- The model learns when to generate each modality based on task requirements
- Training on only ~24K examples enables generalization to out-of-domain tasks

This approach differs from sequential multimodal reasoning (generate all text, then all images) or merged representations by creating tight coupling: each reasoning step may involve both modalities simultaneously.

## Architecture Overview

- **Base Model**: Unified architecture (e.g., Bagel-7B) with shared token vocabulary for text and image patches
- **Reasoning Modes**: Three training modes: text-only, visual-only, interleaved
- **Loss Functions**: Dual objectives combining MSE for image reconstruction and NLL for text prediction
- **Dataset**: ~24K curated traces across four vision-centric task categories with human quality control
- **Evaluation**: Vision-centric benchmarks (SAT, MMVP, VStar, BLINK, CV-Bench) plus out-of-domain generalization tests

## Implementation Steps

**1. Curate High-Quality Interleaved Reasoning Traces**

Create training data by collecting problem-solving sequences where humans solve vision tasks while generating both text thoughts (reasoning steps) and visual annotations (highlights, edits, overlays). Use GPT-4 to generate candidate traces, then human raters filter for coherence and correctness.

```python
# Example structure of an interleaved trace
trace = {
    "task": "locate_object_in_image",
    "steps": [
        {"type": "text", "content": "I need to scan the image systematically"},
        {"type": "image", "content": "<visual_annotation>", "action": "highlight_region"},
        {"type": "text", "content": "The object appears in the upper-left area"},
        {"type": "image", "content": "<zoomed_patch>", "action": "zoom_in"},
    ]
}
# Organize by task category: jigsaw (6K), navigation (6K), search (7K), chart (6K)
```

**2. Tokenize Mixed Modality Sequences**

Represent images as sequences of tokens from an image vocabulary (e.g., using a VQ-VAE codebook). Interleave text and image token sequences into a single stream, enabling standard transformer training. Use special tokens to mark modality boundaries.

```python
# Tokenization of interleaved sequence
text_tokens = tokenizer.encode("I need to locate")  # [512, 1023, 450, ...]
image_tokens = vqvae.encode(image_patch)  # [2048, 2051, 2050, ...]
# Interleave: mark boundaries with modality tokens
interleaved = [512, 1023, 450, MODALITY_IMG, 2048, 2051, MODALITY_TXT, ...]
# Standard next-token prediction loss applies uniformly
```

**3. Implement Dual Loss Training**

Combine reconstruction loss for images (MSE on patch space) with NLL for text tokens. Weight appropriately based on modality mix in batches.

```python
# Dual loss computation
def compute_loss(predictions, targets, modalities):
    text_mask = modalities == MODALITY_TXT
    image_mask = modalities == MODALITY_IMG

    text_loss = F.cross_entropy(predictions[text_mask], targets[text_mask])
    image_loss = F.mse_loss(predictions[image_mask], targets[image_mask])

    return text_loss + lambda_image * image_loss
```

**4. Train on Diverse Vision Tasks**

Fine-tune the base model on the curated dataset using causal language modeling. The model learns to map task descriptions to appropriate reasoning patterns through exposure to diverse problem types.

```python
# Training loop structure
for epoch in range(num_epochs):
    for batch in dataloader:
        traces = batch["interleaved_tokens"]  # Mix of text and image tokens
        loss = compute_loss(model(traces), traces)
        loss.backward()
        optimizer.step()
```

## Practical Guidance

**When to Use**:
- Vision-centric reasoning requiring spatial manipulation (puzzles, navigation, visual search)
- Tasks where text reasoning alone leaves important spatial information implicit
- Scenarios where test-time compute budgeting is critical (interleaved-N=4 > text-only-N=8)

**Hyperparameters**:
- Data curation: Aim for 4-6K high-quality examples per task category
- λ_image: Weight for image reconstruction loss (typically 0.5-1.0)
- Training epochs: 2-3 passes usually sufficient with curriculum (easier tasks first)

**When NOT to Use**:
- Pure text reasoning tasks (language understanding, knowledge QA)
- Tasks without clear visual manipulation requirements
- Settings where latency is critical (image tokenization adds overhead)

**Pitfalls**:
- **Insufficient data curation**: Poor-quality traces teach spurious correlations between text and images
- **Modality imbalance**: Too few image tokens causes visual reasoning to remain underdeveloped
- **Overfitting to task distribution**: The 24K examples must span diverse problem types or generalization fails

**Emergent Properties to Exploit**:
1. **Autonomous mode-switching**: Despite training only on interleaved data, models spontaneously use text-only reasoning when appropriate (~5% of cases), achieving higher accuracy than forced interleaving
2. **Superior scaling under best-of-N**: Interleaved reasoning explores more diverse trajectories, yielding +5-8% gains over text-only under sampling
3. **Novel visual manipulations**: Models generate unseen edits during inference (perspective transforms, inpainting) without explicit training

**Integration Point**: Combine with VLM fine-tuning pipelines—interleaved reasoning complements supervised fine-tuning on domain-specific tasks.

## Reference

arXiv: https://arxiv.org/abs/2510.27492