Masked Autoencoders Are Scalable Vision Learners Arxiv 2111 06377v2

Masked Autoencoders Are Scalable Vision Learners

One-line decision

Use this skill when you want to pre-train a ViT by masking 75% of image patches and reconstructing them, learning rich visual features. Avoid it when contrastive pretraining (CLIP, DINO) is sufficient.

Skill metadata

• Skill type: masked-image-modeling-pretraining
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Pre-train Vision Transformers by masking 75% of image patches and training the model to reconstruct them, learning rich visual features through an efficient self-supervised approach.

Problem signature

• Modality: images with masked patch reconstruction pretraining.
• Data state: ImageNet images for masked autoencoder pretraining.
• Scale regime: ImageNet scale with efficient training.
• Model requirement: ViT encoder + lightweight decoder for masked image modeling.

Use when

• You want efficient self-supervised ViT pretraining.
• You need features from masked image modeling.
• You want to pretrain vision encoders cheaply.

Do not use when

• Contrastive pretraining is preferred.
• You already have pretrained vision encoders.
• The downstream task favors contrastive features.

Required inputs

• pretraining_images: ImageNet for MAE pretraining.
• mae_architecture: ViT encoder + lightweight decoder.
• masking_strategy: Random masking of 75% of patches.

Optional inputs

• downstream_data: Data for fine-tuning evaluation.

Outputs

• mae_encoder: Pre-trained ViT encoder from MAE.
• visual_features: Rich visual features from reconstruction.

Assumptions and prerequisites

• Masking 75% of patches forces meaningful feature learning.
• Reconstruction teaches visual representation.
• MAE is more computationally efficient than contrastive methods.

Procedure

1. Mask image patches Action: Randomly mask 75% of image patches. Why: High masking ratio forces learning. Note: See paper for details.
2. Encode visible patches Action: Process only visible patches with the encoder. Why: Efficient: only 25% of patches processed. Note: See paper for details.
3. Reconstruct masked patches Action: Decode to reconstruct the original image. Why: Reconstruction teaches visual features. Note: See paper for details.
4. Use encoder for downstream Action: Discard decoder; use encoder features. Why: Encoder features are the goal. Note: See paper for details.

Parameters to set

• mask_ratio — Role: Fraction of patches masked. How to set: 75% for optimal learning. Default/range: 75%. Effect: Higher ratio forces more difficult reconstruction.
• decoder_depth — Role: Depth of the reconstruction decoder. How to set: Lightweight (2-4 layers). Default/range: 4 layers. Effect: Lightweight decoder makes training efficient.

Validation checks

• Fine-tuned MAE should achieve strong ImageNet accuracy.
• 75% masking should outperform lower ratios.
• MAE should be more efficient than contrastive pretraining.

Failure modes

• MAE features may differ from contrastive features.
• Fine-tuning is typically needed (unlike CLIP zero-shot).
• Reconstruction quality may not directly predict downstream quality.

Adaptation notes for VLM training

• MAE provides EVA-style initialization for VLM vision encoders.
• MAE + CLIP combined provides complementary features.
• The efficient pretraining approach scales well.

Implementation notes

• Use the MAE codebase for pretraining.
• Process only visible patches for efficiency.
• Compare to DINO and CLIP pretraining.

Evidence from the paper

• MAE pre-trains ViTs by reconstructing 75% masked patches.
• The approach is significantly more efficient than contrastive methods.
• MAE features are strong after fine-tuning.
• High masking ratio (75%) is surprisingly effective.

Source paper

• Title: Masked Autoencoders Are Scalable Vision Learners
• Year: 2022
• Venue: CVPR
• Paper ID: arxiv-2111.06377v2
• URL: http://arxiv.org/abs/2111.06377v2
• arXiv ID: 2111.06377v2