# Uniworld Semantic Vision > Combine semantic encoders from multimodal LLMs with contrastive learning to create unified high-resolution encoders for both visual understanding and generation tasks without relying on VAE compression. - Skill: `adu2021/uniworld-semantic-vision` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/uniworld-semantic-vision` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/uniworld-semantic-vision/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/uniworld-semantic-vision --- # UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation ## Core Concept UniWorld proposes using semantic encoders—derived from powerful multimodal language models like GPT-4o—as replacements for traditional Variational Autoencoders (VAEs) in vision tasks. The insight is that semantic encoders capture meaningful visual features that support both understanding (image classification, captioning) and generation (image synthesis, manipulation) tasks. The framework combines semantic feature extraction from multimodal models with contrastive learning, enabling a unified architecture for diverse vision tasks. By leveraging pretrained semantic features rather than learning from scratch, UniWorld achieves strong performance with only 2.7M training samples and provides fully open-source access to model weights and training scripts. ## Architecture Overview - **Semantic Feature Extraction**: Leverage multimodal LLM encoders (e.g., GPT-4o-Image) rather than VAEs - **Contrastive Learning**: Apply contrastive objectives to align visual features with semantic meaning - **Unified Encoder Design**: Single architecture supporting understanding and generation - **High-Resolution Support**: Preserve fine details without VAE quantization losses - **Modular Integration**: Compatible with existing vision models and generation frameworks - **Open-Source Foundation**: Provide weights, training scripts, and datasets for reproducibility ## Implementation The following steps outline how to implement semantic encoders for unified vision tasks: 1. **Extract semantic embeddings** - Use multimodal LLM to compute semantic representations of images 2. **Prepare contrastive training data** - Create positive/negative image pairs with semantic similarity labels 3. **Train semantic encoder** - Optimize encoder using contrastive learning objectives 4. **Adapt for understanding** - Fine-tune encoder for image classification, captioning, etc. 5. **Adapt for generation** - Use encoder as backbone for diffusion or autoregressive generation 6. **Evaluate on benchmarks** - Test across understanding and generation tasks ```python import torch import torch.nn as nn import torch.nn.functional as F from typing import Tuple, List class SemanticEncoder(nn.Module): """Semantic encoder for vision-language tasks.""" def __init__(self, embedding_dim: int = 768, hidden_dim: int = 2048): super().__init__() self.embedding_dim = embedding_dim # Feature extraction backbone (e.g., Vision Transformer) self.backbone = nn.Sequential( nn.Linear(3 * 224 * 224, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, embedding_dim) ) # Projection head for contrastive learning self.projection_head = nn.Sequential( nn.Linear(embedding_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, embedding_dim) ) def forward(self, images: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: """Extract semantic embeddings from images.""" # Flatten images if needed batch_size = images.shape[0] if images.dim() == 4: images = images.reshape(batch_size, -1) # Extract features features = self.backbone(images) # Project to contrastive space embeddings = self.projection_head(features) embeddings = F.normalize(embeddings, p=2, dim=1) return features, embeddings class ContrastiveLearner: """Train semantic encoder with contrastive objectives.""" def __init__(self, encoder: SemanticEncoder, temperature: float = 0.07): self.encoder = encoder self.temperature = temperature def contrastive_loss(self, embeddings_i: torch.Tensor, embeddings_j: torch.Tensor) -> torch.Tensor: """Compute NT-Xent (normalized temperature-scaled cross entropy) loss.""" batch_size = embeddings_i.shape[0] # Concatenate embeddings embeddings = torch.cat([embeddings_i, embeddings_j], dim=0) # Compute similarity matrix similarity_matrix = torch.mm(embeddings, embeddings.t()) similarity_matrix = similarity_matrix / self.temperature # Create labels (matching pairs should be similar) mask = torch.eye(batch_size, dtype=torch.bool) mask = torch.cat([torch.cat([torch.zeros_like(mask), mask], dim=1), torch.cat([mask, torch.zeros_like(mask)], dim=1)], dim=0) # Compute loss labels = torch.arange(batch_size) labels = torch.cat([labels, labels + batch_size], dim=0) loss = F.cross_entropy(similarity_matrix, labels) return loss def train_step(self, images_i: torch.Tensor, images_j: torch.Tensor, optimizer: torch.optim.Optimizer) -> float: """Execute one training step.""" _, embeddings_i = self.encoder(images_i) _, embeddings_j = self.encoder(images_j) loss = self.contrastive_loss(embeddings_i, embeddings_j) optimizer.zero_grad() loss.backward() optimizer.step() return loss.item() class UnifiedVisionModel: """Unified model for understanding and generation.""" def __init__(self, semantic_encoder: SemanticEncoder): self.encoder = semantic_encoder self.understanding_head = nn.Linear(768, 1000) # Classification self.generation_head = nn.Linear(768, 256 * 256 * 3) # Pixel reconstruction def understand(self, images: torch.Tensor) -> torch.Tensor: """Perform image understanding (e.g., classification).""" features, _ = self.encoder(images) logits = self.understanding_head(features) return logits def generate(self, latents: torch.Tensor) -> torch.Tensor: """Perform image generation from latent codes.""" reconstruction = self.generation_head(latents) # Reshape to image dimensions batch_size = reconstruction.shape[0] images = reconstruction.reshape(batch_size, 3, 256, 256) return torch.sigmoid(images) def reconstruction_loss(self, images: torch.Tensor) -> torch.Tensor: """Compute reconstruction loss for generation.""" features, _ = self.encoder(images) reconstructed = self.generate(features) # Resize for comparison target_size = reconstructed.shape[2:] images_resized = F.interpolate(images, size=target_size, mode='bilinear') loss = F.mse_loss(reconstructed, images_resized) return loss class SemanticEncoderTrainer: def __init__(self, encoder: SemanticEncoder, device: str = 'cpu'): self.encoder = encoder.to(device) self.device = device self.learner = ContrastiveLearner(encoder) def train(self, train_loader, num_epochs: int = 10, learning_rate: float = 0.001): """Train semantic encoder on image dataset.""" optimizer = torch.optim.Adam(self.encoder.parameters(), lr=learning_rate) for epoch in range(num_epochs): total_loss = 0.0 for images_i, images_j in train_loader: images_i = images_i.to(self.device) images_j = images_j.to(self.device) loss = self.learner.train_step(images_i, images_j, optimizer) total_loss += loss avg_loss = total_loss / len(train_loader) print(f"Epoch {epoch}: Loss = {avg_loss:.4f}") return self.encoder ``` ## Practical Guidance **Architecture choices:** - **Feature extractor**: Use Vision Transformer (ViT) for modern architectures; ResNet for efficiency - **Embedding dimension**: 768 is standard (matches BERT); 1024+ for higher-capacity models - **Projection head**: Use simple MLP (2-3 layers); avoid overly complex projections **Training strategies:** - **Contrastive temperature**: 0.07 is standard; adjust to 0.05-0.10 based on dataset scale - **Batch size**: 256-512 recommended for stable contrastive learning - **Data augmentation**: Apply strong augmentations to image pairs for robustness - **Epochs**: 10-50 typically sufficient; monitor validation metrics to detect overfitting **When to use:** - Unified vision systems handling both understanding and generation - Fine-grained visual reasoning where semantic features are valuable - Building vision-language models with alignment requirements - Research on visual feature learning and representation quality **When NOT to use:** - Task-specific models where specialized architectures outperform generalists - Extremely high-resolution images where computational overhead is prohibitive - Domain where domain-specific feature extraction is critical (medical imaging, etc.) - Streaming applications requiring real-time inference **Common pitfalls:** - **Weak augmentation**: Insufficient data augmentation hurts contrastive learning effectiveness - **Temperature miscalibration**: Too high temperature makes loss uninformative; too low causes optimization instability - **Feature collapse**: Encoder may learn trivial solutions; use momentum contrast or other regularization - **Dataset bias**: Models learn dataset-specific features; validate on out-of-domain data - **Scalability**: Contrastive learning requires large batches; may not scale to very small datasets ## Reference UniWorld demonstrates that semantic encoders from multimodal models can effectively replace VAEs for vision tasks. The framework is fully open-source with model weights, training scripts, and 2.7M sample training dataset available. The approach unifies understanding and generation in a single architecture without specialized VAE quantization. Original paper: "UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation" (arxiv.org/abs/2506.03147)