# Vfm Visual Tokenizer > Use frozen vision foundation models like DINOv2 and CLIP as image tokenizers for autoregressive generation. Region-adaptive quantization identifies semantically coherent areas and reduces redundancy. Achieves 256-token encoding (vs. 576), 3× AR model speedup, state-of-the-art 1.36 gFID on ImageNet while eliminating classifier-free guidance. - Skill: `adu2021/vfm-visual-tokenizer` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/vfm-visual-tokenizer` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/vfm-visual-tokenizer/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/vfm-visual-tokenizer --- # VFMTok: Foundation Model-Based Visual Tokenization for Efficient AR Image Generation Image generation typically requires tokenizing images into compact codes for autoregressive modeling. Learned tokenizers (VAE variants) need training and may lose semantic information. VFMTok leverages frozen vision foundation models—already trained on massive datasets to understand images semantically—as tokenizers. By adding region-adaptive quantization (identifying semantically coherent clusters rather than fixed grids), the approach reduces tokens by 55% (256 vs. 576), accelerates AR model training by 3×, and eliminates classifier-free guidance while maintaining state-of-the-art quality. The key insight is that foundation models already extract meaningful features; you only need to make them quantize-friendly by identifying semantic regions and learning lightweight codebooks, not building entire new encoders. ## Core Concept VFMTok operates through a frozen pipeline: 1. **Frozen VFM Encoder**: Extract multi-level features from DINOv2 or CLIP (layers 6, 12, 18, 24) 2. **Deformable Attention Sampler**: Use learnable anchor queries to identify semantically coherent regions (adaptive quantization) 3. **Lightweight Codebook**: Learn small discrete vocabulary (16K codes) mapping regions to indices 4. **Semantic Reconstruction**: Dual objectives—pixel-level fidelity and VFM feature preservation The frozen foundation model provides semantic understanding; the learnable components focus purely on efficient discretization. ## Architecture Overview - **Frozen VFM Backbone**: DINOv2-L or CLIP-L (no gradient flow) - **Multi-level Feature Extraction**: Concatenate features from 4 layers (dims: 256→512→768→1024) - **Deformable Attention Module**: Learnable anchor queries with deformable sampling to find semantic regions - **Codebook Vector Quantizer (VQ)**: Maps regional features to discrete codes (size: 16384, dim: 12) - **Lightweight Decoder**: Learnable ViT-style decoder reconstructing pixels from codes - **Dual Loss**: Pixel-level L2 + VFM feature reconstruction via cosine distance - **Positional Embeddings**: 2D spatial encoding for anchor positions ## Implementation The following demonstrates region-adaptive quantization and the tokenization pipeline: ```python import torch import torch.nn as nn import torch.nn.functional as F from typing import Tuple class DeformableAttentionSampler(nn.Module): """Learnable attention for identifying semantically coherent regions.""" def __init__(self, feature_dim: int = 1024, num_anchors: int = 256, num_heads: int = 8): super().__init__() self.feature_dim = feature_dim self.num_anchors = num_anchors self.num_heads = num_heads # Learnable anchor positions (where to sample from feature maps) self.anchor_queries = nn.Parameter(torch.randn(1, num_anchors, feature_dim)) # Deformable offset regression (predicts sampling locations) self.offset_regression = nn.Linear(feature_dim, 2) # 2D spatial offsets # Multi-head cross-attention self.cross_attention = nn.MultiheadAttention( feature_dim, num_heads, batch_first=True ) # Positional embeddings for anchor positions self.pos_embedding = nn.Embedding(num_anchors, feature_dim) def forward(self, features: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: """ Sample semantically coherent regions using deformable attention. Args: features: (batch, seq_len, feature_dim) where seq_len = H*W from VFM Returns: regional_features: (batch, num_anchors, feature_dim) anchor_positions: (batch, num_anchors, 2) sampled 2D positions """ batch_size = features.shape[0] # Add position embeddings to anchor queries positions = torch.arange(self.num_anchors, device=features.device) pos_emb = self.pos_embedding(positions).unsqueeze(0).expand(batch_size, -1, -1) anchors_with_pos = self.anchor_queries.expand(batch_size, -1, -1) + pos_emb # Predict offsets for each anchor (adaptive sampling locations) offsets = torch.tanh(self.offset_regression(anchors_with_pos)) * 0.5 # Bounded offsets anchor_positions = offsets # (batch, num_anchors, 2) normalized [-0.5, 0.5] # Deformable sampling: index features using predicted positions # In practice, use grid_sample or custom indexing sampled_features = self._deformable_sample(features, anchor_positions) # Cross-attention: focus on high-confidence regions attended, _ = self.cross_attention( query=anchors_with_pos, key=sampled_features, value=sampled_features ) return attended, anchor_positions def _deformable_sample(self, features: torch.Tensor, positions: torch.Tensor) -> torch.Tensor: """Bilinear sampling from feature map given normalized positions.""" # features: (batch, H*W, feature_dim) # positions: (batch, num_anchors, 2) normalized to [-0.5, 0.5] batch_size, seq_len, feature_dim = features.shape num_anchors = positions.shape[1] h = w = int(seq_len ** 0.5) # Assume square feature map # Convert normalized positions to pixel coordinates pixel_x = (positions[:, :, 0] + 0.5) * (w - 1) pixel_y = (positions[:, :, 1] + 0.5) * (h - 1) # Bilinear interpolation (grid_sample style) grid = torch.stack([pixel_x / (w - 1) * 2 - 1, pixel_y / (h - 1) * 2 - 1], dim=-1) # Reshape features to spatial format for grid_sample features_spatial = features.view(batch_size, h, w, feature_dim).permute(0, 3, 1, 2) sampled = F.grid_sample( features_spatial.float(), grid.unsqueeze(1).float(), # (batch, 1, num_anchors, 2) align_corners=True, mode='bilinear' ) return sampled.permute(0, 2, 3, 1).squeeze(2) # (batch, num_anchors, feature_dim) class VectorQuantizer(nn.Module): """Discrete codebook for quantizing regional features.""" def __init__(self, codebook_size: int = 16384, feature_dim: int = 12, beta: float = 0.25): super().__init__() self.codebook_size = codebook_size self.feature_dim = feature_dim self.beta = beta # Codebook: learnable embeddings self.embedding = nn.Embedding(codebook_size, feature_dim) self.embedding.weight.data.uniform_(-1 / codebook_size, 1 / codebook_size) def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: """ Quantize features to discrete codes. Args: x: (batch, num_regions, feature_dim) Returns: quantized: (batch, num_regions, feature_dim) - discretized features loss: scalar quantization loss indices: (batch, num_regions) - discrete code indices """ # Compute distances to all codebook entries # (batch, num_regions, feature_dim) @ (feature_dim, codebook_size) distances = torch.cdist(x, self.embedding.weight) # (batch, num_regions, codebook_size) # Find nearest codebook entries indices = distances.argmin(dim=-1) # (batch, num_regions) quantized = self.embedding(indices) # (batch, num_regions, feature_dim) # Vector quantization loss (commitment loss) loss = F.mse_loss(x.detach(), quantized) + self.beta * F.mse_loss(x, quantized.detach()) return quantized, loss, indices class VFMTokenizer(nn.Module): """Complete frozen VFM-based image tokenizer.""" def __init__(self, vfm_model_name: str = "facebook/dino-vitl14", num_anchor_regions: int = 256, codebook_size: int = 16384): super().__init__() self.num_anchor_regions = num_anchor_regions # Frozen VFM encoder self.vfm = None # Load pretrained model from vfm_model_name self.vfm_feature_dim = 1024 # Dimension of concatenated features # Make VFM frozen for param in self.vfm.parameters(): param.requires_grad = False # Learnable tokenization components self.deformable_sampler = DeformableAttentionSampler( feature_dim=self.vfm_feature_dim, num_anchors=num_anchor_regions, num_heads=8 ) self.quantizer = VectorQuantizer( codebook_size=codebook_size, feature_dim=12 # Small codebook feature dimension ) # Lightweight decoder (reconstructs images from codes) self.decoder = nn.Sequential( nn.Linear(12, 64), nn.ReLU(), nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 3) # RGB output ) def forward(self, image: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: """ Tokenize image into discrete codes. Args: image: (batch, 3, 336, 336) RGB image Returns: codes: (batch, num_anchor_regions) integer indices loss: quantization + reconstruction loss reconstructed: (batch, 3, 336, 336) decoded image """ # Extract multi-level features from frozen VFM with torch.no_grad(): # Get features from multiple layers feat_layer6 = self.vfm.get_layer_output(image, layer_idx=6) # (batch, seq_len, 256) feat_layer12 = self.vfm.get_layer_output(image, layer_idx=12) # (batch, seq_len, 512) feat_layer18 = self.vfm.get_layer_output(image, layer_idx=18) # (batch, seq_len, 768) feat_layer24 = self.vfm.get_layer_output(image, layer_idx=24) # (batch, seq_len, 1024) # Concatenate across layers features = torch.cat([ F.interpolate(feat_layer6, size=feat_layer24.shape[1:], mode='nearest'), F.interpolate(feat_layer12, size=feat_layer24.shape[1:], mode='nearest'), F.interpolate(feat_layer18, size=feat_layer24.shape[1:], mode='nearest'), feat_layer24 ], dim=-1) # (batch, seq_len, 1024) # Identify semantic regions via deformable attention regional_features, anchor_positions = self.deformable_sampler(features) # regional_features: (batch, num_anchor_regions, 1024) # Project to codebook dimension projected = F.linear(regional_features, torch.randn(12, 1024)) # (batch, num_anchor_regions, 12) # Quantize to discrete codes quantized, quant_loss, codes = self.quantizer(projected) # codes: (batch, num_anchor_regions) # Reconstruct image from quantized codes reconstructed = self.decoder(quantized) # (batch, num_anchor_regions, 3) # Reshape to image (simple approach: map regions to spatial grid) # In practice, use more sophisticated spatial reconstruction grid_h = grid_w = int(self.num_anchor_regions ** 0.5) reconstructed_image = reconstructed.view(-1, grid_h, grid_w, 3).permute(0, 3, 1, 2) reconstructed_image = F.interpolate(reconstructed_image, size=(336, 336), mode='bilinear') # Total loss: quantization + reconstruction recon_loss = F.mse_loss(reconstructed_image, image) total_loss = quant_loss + recon_loss return codes, total_loss, reconstructed_image def train_vfm_tokenizer(model: VFMTokenizer, train_loader, optimizer: torch.optim.Optimizer, vfm_feature_loss_weight: float = 0.5, num_epochs: int = 50): """Train tokenizer with frozen VFM backbone.""" for epoch in range(num_epochs): total_loss = 0 for batch_idx, images in enumerate(train_loader): optimizer.zero_grad() # Forward pass codes, loss, reconstructed = model(images) # Optional: add VFM feature reconstruction loss # (ensure quantized features align with VFM semantics) # This preserves the semantic meaning of codes total_loss_iter = loss total_loss_iter.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() total_loss += total_loss_iter.item() avg_loss = total_loss / len(train_loader) print(f"Epoch {epoch + 1}/{num_epochs}: Loss {avg_loss:.6f}") if (epoch + 1) % 10 == 0: # Evaluate: compute token count and reconstruction quality sample_codes, _, sample_recon = model(next(iter(train_loader))) avg_tokens = (sample_codes >= 0).sum(dim=1).float().mean().item() print(f" Average tokens per image: {avg_tokens:.0f} (target: 256)") ``` This implementation shows how to leverage frozen foundation models with learnable quantization for efficient tokenization. ## Practical Guidance | Component | Parameter | Recommendation | |-----------|-----------|-----------------| | **VFM Backbone** | Model | DINOv2-L or CLIP-L (both ~300M params) | | **Anchor Regions** | num_anchor_regions | 256 (16×16 grid equivalent with semantic grouping) | | **Codebook Dimension** | feature_dim | 12 (small to compress; larger hurts AR efficiency) | | **Codebook Size** | codebook_size | 16384 (2^14) for expressive tokenization | | **Quantization Loss Weight** | beta | 0.25-0.5 | | **Reconstruction Loss** | pixel-level | L2 (MSE) sufficient; skip perceptual loss (VFM is semantic) | | **Learning Rate** | optimizer | 1e-3 (learnable components only) | ### When to Use VFMTok - **Autoregressive image generation**: Reducing token count (256 vs. 576) directly improves AR model speed and quality - **Efficient image synthesis**: 3× speedup in AR model training is critical for resource-constrained settings - **Preserving semantic understanding**: Foundation models already encode semantic information; leverage it - **Avoiding classifier-free guidance**: Semantic tokens mean unconditional generation less noisy; CFG not needed - **Transfer learning**: VFM semantics transfer across domains; tokenizer generalization excellent - **Multi-task vision**: One tokenizer serves generation, compression, and analysis tasks ### When NOT to Use - **Domain-specific image styles**: If your domain differs significantly from VFM training data (medical, satellite), learned tokenizers may adapt better - **Real-time encoding**: Frozen models still require forward passes; not suitable for <10ms encoding latency - **Extreme compression** (<100 tokens): Region-adaptive quantization has limits; may require learned quantization schedules - **Fine-grained detail preservation** (>4K resolution): VFM patch size limits spatial resolution; fine details lost - **Adversarial robustness**: Frozen foundation models may inherit VFM vulnerabilities; learned components don't increase robustness ### Common Pitfalls 1. **Codebook Collapse**: If codebook_size is too small (e.g., <1000), many regions map to same codes. Increase to 16K+, or add codebook diversity losses. 2. **Anchor Dropout**: Deformable sampler may ignore hard-to-learn regions. Add explicit region coverage loss: encourage non-overlapping anchor sampling. 3. **VFM Feature Mismatch**: If reconstruction loss dominates and ignores VFM semantics, add auxiliary loss: F.cosine_similarity(codes, vfm_features) > 0.7. Weight: 0.5. 4. **Spatial Coherence**: Simple grid-based reconstruction loses spatial relationships. Use learned spatial decoder: map codes → 2D spatial embeddings, then CNN upsampling. 5. **Frozen Backbone Underutilization**: If you freeze the entire VFM, you miss the opportunity to fine-tune layer-specific extraction. Consider freezing only bottom 50% of VFM, fine-tune top layers. ## Reference Chen, L., Zhou, Y., et al. (2025). Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation. *arXiv preprint arXiv:2507.08441*. Available at: https://arxiv.org/abs/2507.08441