Canvit Active Vision Foundation Model

The Problem Statement

What is this problem?

Active vision—the problem of intelligently selecting where to look in an image or scene—has historically been studied as a specialized subtask, separate from general vision foundation models. Most vision models (ResNets, ViTs, DINOv2) process entire images uniformly, treating the viewing problem as solved. But in robotics, surveillance, or embodied AI, agents must learn how and where to look given computational constraints.

Before CanViT, no scalable, general-purpose architecture existed for active vision that could:

• Learn from diverse viewpoints (not task-specific gaze patterns)
• Adapt to new tasks via transfer learning (foundation model paradigm)
• Scale to large scenes without memory explosion
• Work across embodied AI domains (robotics, surveillance, navigation)

Why is this important?

Most real-world vision systems are resource-constrained: robotics arms, mobile agents, or edge deployment. Selective attention is fundamental—you can't process high-resolution images of entire environments in real-time. Yet active vision research was fragmented, domain-specific, and lacked unified foundation models.

CanViT establishes that foundation models can be built for active vision, opening a new research direction with broad applicability.

What existing approaches are inadequate?

• Standard ViTs/CNNs: Process full images, waste computation on irrelevant regions
• Policy-specific active vision models: Hand-designed for particular tasks (object detection, semantic segmentation); don't transfer
• Attention mechanisms: Soft spatial attention, but don't model sequential eye movements or sampling patterns
• No foundation model paradigm: Active vision lacked the pretraining scale and transfer-learning infrastructure of generic vision

The Paradigm Shift: Active-Vision Foundation Models

What new way of thinking does CanViT introduce?

Active-Vision Foundation Models (AVFM) reframe vision as a sequential decision problem: given computational constraints, what regions should the model examine, and in what order? Foundation models enable transfer of learned visual-attentional strategies to new domains.

Key conceptual innovations:

1. Retinotopic representation: Process visual input as a sequence of foveated glimpses, mimicking biological vision. Each glimpse is high-resolution near a fixation point, lower-resolution periphery.

2. Decoupled architecture: Separate the "thinking" component (processing individual glimpses) from the "memory" component (maintaining scene understanding across glimpses). This is different from end-to-end black-box processing.

3. Canvas representation: Maintain a latent "canvas"—a global working memory of the scene—that accumulates information across glimpses without storing raw pixels.

4. Scene-relative coordinates: Use rotary positional embeddings anchored to scene coordinates (not image coordinates), enabling generalization to scenes of different sizes.

How does this reframe active vision?

Before CanViT's paradigm:

• Active vision was a specialized subtask ("add attention to my existing model")
• Each task required hand-designed gaze policies
• No transfer learning infrastructure (each domain started from scratch)
• Scaling to large scenes was hard (memory and compute both grow with scene size)

After CanViT's paradigm:

• Active vision is a foundational problem class deserving general-purpose models
• Foundation pretraining learns universal visual-attentional strategies
• Transfer learning enables rapid adaptation to new active-vision tasks
• Decoupled architecture (glimpse processor + canvas memory) scales to large scenes

Core Architecture & Innovation

Retinotopic Vision Transformer (RoViT):

Process input as foveated glimpses rather than full images. Each glimpse has:

• High-resolution center (fovea): ~224×224 pixels near fixation
• Lower-resolution periphery: contextual information
• Sparse sampling rather than dense processing

Canvas Attention Mechanism:

Key innovation: asymmetric cross-attention between the ViT backbone and scene canvas:

# Canvas Attention enables efficient interaction without overhead
# Backbone updates canvas selectively; canvas-to-backbone projection is minimal

def canvas_attention(glimpse_features, canvas_state):
    """
    Asymmetric cross-attention:
    - Backbone processes glimpse → generates updates
    - Updates flow to canvas (dense)
    - Canvas queries backbone minimally (sparse)
    Result: low-latency sequential inference
    """
    # Backbone: dense ViT processing on single glimpse
    glimpse_embedding = transformer_backbone(glimpse_features)

    # Canvas update: add glimpse information to scene canvas
    updated_canvas = canvas_state.update(glimpse_embedding)

    # Minimal feedback: canvas → backbone via position encoding only
    # (No expensive cross-attention layers)
    positional_update = scene_relative_rope(updated_canvas, current_fixation)

    return updated_canvas, positional_update

Scene-Relative RoPE Binding:

Position embeddings are anchored to scene coordinates, not image coordinates. This enables:

• Generalization across different scene sizes
• Composable spatial reasoning (understand where in the scene you're looking)
• Transfer to new scenes with different layouts

Label-Free Pretraining via Dense Latent Distillation:

Instead of supervised pretraining, CanViT uses:

• Random glimpse sequences (1B glimpses on ImageNet-21k)
• Distillation target: DINOv3 embeddings (self-supervised foundation model)
• Learn to reconstruct scene-wide DINOv3 embeddings from random glimpse samples

This is fundamentally unsupervised and policy-agnostic—doesn't require labeled gaze data or task-specific supervision.

Founding Experiments & Empirical Evidence

Benchmark 1: ADE20K Semantic Segmentation (frozen probes)

• Task: Single low-resolution glimpse, predict semantic segmentation of full scene
• CanViT: 38.5% mIoU
• Prior active-vision baseline: 27.6% mIoU
• Efficiency: 19.5× fewer FLOPs than baselines

Surprising finding: With just one glimpse and frozen linear probes, CanViT outperforms all prior active-vision methods. This validates that foundation pretraining learns transferable visual priors.

Benchmark 2: ImageNet-1k Classification

• Task: Classify images via random glimpse sequences
• CanViT: 81.2% top-1 accuracy with frozen teacher probes
• Validation: Frozen probes confirm learned representations transfer, not task-specific memorization

Benchmark 3: Pretraining Scale

• Data: 13.2 million ImageNet-21k scenes
• Glimpses: 1 billion random glimpses
• Compute: 166 hours on single H100 GPU
• Conclusion: Efficient pretraining enables foundation-model scale for active vision

Key ablations:

• Remove retinotopic structure: Canvas accuracy drops (spatial coherence matters)
• Remove Canvas Attention (use standard cross-attention): Latency and memory explode
• Remove scene-relative RoPE: Generalization to different scene sizes fails

Vocabulary & Foundational Concepts

Pre vs. Post CanViT

Before CanViT:

• Active vision: domain-specific models, hand-crafted for robotics/surveillance/etc.
• No transfer learning: each task required training from scratch
• Scalability issues: large scenes required expensive attention mechanisms
• No foundation model: active vision lacked infrastructure of generic vision models
• Limited diversity: most systems optimized for single task or domain

After CanViT:

• Active vision: recognized as a foundational problem deserving general models
• Transfer learning enabled: pretrain once, adapt to robotics, surveillance, embodied AI
• Architectural innovations: Canvas Attention scales to arbitrary scene sizes
• Foundation model paradigm: AVFM enables efficient knowledge transfer
• Broad applicability: demonstrated on segmentation, classification, and various scene types

Opened Research Directions

1. Task-specific adaptation: How to fine-tune AVFM for specific downstream active-vision tasks (robotics control, real-time tracking)?
2. Sequential decision-making: Can AVFM predict which glimpse to examine next given a query or goal?
3. Hierarchical active vision: How to combine AVFM with higher-level task planning (what goal requires what glimpses)?
4. Real-time deployment: How to optimize CanViT for edge devices or embedded systems?
5. Embodied integration: Can AVFM drive active agents in simulation or real robots without modification?
6. Multi-agent coordination: Do AVFM strategies generalize to multi-agent perception (multiple cameras, multiple agents)?
7. Continual learning: Can AVFM update on new scenes without catastrophic forgetting?
8. Interpretability of gaze: What makes AVFM select certain glimpses—can we understand its attention patterns?

When to Use the Active-Vision Paradigm

AVFM is appropriate when:

• Your system has computational constraints (can't process full-resolution images)
• Selective attention is necessary (robot arm, mobile agent, edge device)
• Generalization across domains matters (transfer learning efficiency)
• You need to scale beyond single-image models to multi-glimpse understanding
• Embodied AI or robotics applications

Examples:

• Robot manipulation (choose where to look given manipulation task)
• Surveillance systems (selective attention on crowded scenes)
• Autonomous navigation (active perception for navigation)
• Accessibility systems (selective highlighting of relevant scene regions)
• Medical imaging (focus analysis on relevant anatomical regions)

AVFM is NOT appropriate when:

• You have abundant compute for dense processing
• Single-glimpse understanding suffices (static image analysis)
• Uniform processing is preferable (no spatial structure)

Limitations & Caveats

Current limitations:

• CanViT trained on 2D scenes; 3D scenes or dynamic video untested
• Retinotopic representation assumes foveal structure exists
• Dense latent distillation depends on quality of DINO embeddings

Open questions:

• Does AVFM generalize to video (temporal sequences of glimpses)?
• How does performance degrade with extreme resource constraints?
• Can CanViT predict human visual attention patterns?

Related Mechanistic Questions

Insights CanViT enables:

• What makes good glimpses? Analysis of which spatial regions AVFM prioritizes
• How does Canvas Memory accumulate? Interpretability of latent canvas updates
• Does AVFM mimic biological vision? Comparison to human gaze patterns
• What tasks benefit from active vision? Analysis of when retinotopic processing helps

When to Apply Active-Vision Thinking

Reference

Paper: https://arxiv.org/abs/2603.22570 Related field: Active vision, embodied AI, foundation models Related work: MAML (paradigm-shifting foundation), Transformers (architecture innovation)