# Learning Visual Priors LLM Pretraining

> Decompose visual priors into perception and reasoning components, each optimized by distinct data types: reasoning from code/math corpora, perception from diverse modality-rich sources. Use to construct efficient VLM pretraining pipelines balancing multimodal perception with reasoning capability.

- Skill: `adu2021/learning-visual-priors-llm-pretraining` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/learning-visual-priors-llm-pretraining`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/learning-visual-priors-llm-pretraining/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/learning-visual-priors-llm-pretraining

---


# Learning Visual Priors Before Seeing: Optimized VLM Pretraining

This research decomposes how Vision-Language Models develop visual understanding during text-only pretraining, identifying that perception and reasoning components arise from distinct data sources. By optimizing pretraining mixtures accordingly, practitioners can achieve competitive multimodal performance with reduced visual fine-tuning data.

## Core Architecture

- **Perception component**: Emerges from diverse corpora (web text, books, scientific papers)
- **Reasoning component**: Scales with reasoning-focused data (code, mathematics)
- **Mixture optimization**: 60% reasoning + 15% visual-diverse content achieves balanced tradeoff
- **Controlled experiments**: 100+ systematic experiments identifying component origins

## Implementation Steps

Design pretraining data mixture based on component analysis:

```python
# Construct optimized data mixture for VLM pretraining
from vlm_mixture import DataMixture, PerceptionReasoningOptimizer

# Define component-aligned data sources
data_config = {
    "reasoning_content": {
        "code": 0.30,           # Python, JavaScript, SQL
        "mathematics": 0.20,    # Proofs, derivations, problem-solving
        "scientific": 0.10      # Academic papers with logical flow
    },
    "perception_content": {
        "web_text": 0.15,       # Diverse descriptions, narratives
        "books": 0.10,          # Rich contextual language
        "structured": 0.15      # Tables, lists, captions (visual context)
    }
}

mixture = DataMixture(
    config=data_config,
    total_tokens=1_000_000_000,  # 1T tokens
    adapter_architecture="vision_encoder"
)
```

Validate mixture quality across model scales:

```python
# Test across model sizes (340M-13B) with consistent architecture
from vlm_mixture import MixtureValidator

validator = MixtureValidator(
    model_scales=[340e6, 1.3e9, 7e9, 13e9],
    vision_encoder="standard",  # fixed vision backbone
    evaluation_tasks=[
        "image_captioning",
        "vqa",
        "scene_understanding",
        "reasoning"
    ]
)

results = validator.evaluate(mixture)
```

## Practical Guidance

**When to use this approach:**
- Building Vision-Language Models from scratch
- Optimizing data mixture for target capability balance (perception vs. reasoning)
- Resource-constrained settings where efficient pretraining matters
- Adapting to new vision encoders or modalities

**When NOT to use:**
- Fine-tuning existing VLMs (focus on supervised adaptation instead)
- Purely text-based models where visual priors unnecessary
- Domains with abundant labeled multimodal data (supervised fine-tuning superior)

**Hyperparameter considerations:**
- **Reasoning ratio (30-40%)**: Increase for code-heavy applications; decrease for perception-focused tasks
- **Visual-diverse ratio (15-25%)**: Higher values improve cross-domain generalization; lower values strengthen in-domain performance
- **Adapter architecture**: Standard vision encoder works well; test against task-specific encoders for domain transfer
- **Token budget**: 1T tokens shown optimal; scale down to 100B for efficient prototyping

## Key Findings

**Separable components:**
- Perception: Diffusely distributed across diverse corpora, no single source dominates
- Reasoning: Tightly coupled to code and mathematics, transferable across tasks
- Transfer properties: Reasoning transfers reliably; perception requires broader exposure

**Optimal mixture (mix6):**
- 33.3% overall ranking across 8 evaluation metrics
- Balanced performance on perception and reasoning tasks
- 60% reasoning + 15% visual content + 25% standard text

## Architecture Notes

The Platonic Representation Hypothesis suggests unified world models emerge across modalities. This work supports that finding empirically, showing perception and reasoning separate naturally during pretraining without explicit supervision.

## References

Research builds on understanding of data scaling in language models and vision-language pretraining principles.