Jiachen-T-Wang Sigmoid Loss for Language Image Pre-Training
One-line decision
Use this skill when you want a contrastive loss that handles noisy image-text pairs better by using sigmoid instead of softmax. Avoid it when you are already using softmax contrastive loss successfully on clean data.
Skill metadata
- Skill type: loss-function-for-noisy-data
- Paper kind: operational-method
- Actionability: high
- Evidence quality: full_paper
Goal
Replace the standard softmax contrastive loss in CLIP with a sigmoid-based loss (SigLIP) that operates on independent image-text pairs rather than requiring a softmax over the full batch, enabling better handling of noisy data and more efficient training.
Problem signature
- Modality: image-text pairs with sigmoid-based contrastive training.
- Data state: web-crawled image-text pairs that may contain noise; the sigmoid loss is more robust to false negatives.
- Scale regime: billions of image-text pairs; the sigmoid loss enables larger effective batch sizes.
- Model requirement: Dual-encoder (ViT + text transformer) trained with sigmoid loss instead of softmax contrastive loss.
Use when
- Your image-text training data contains false negatives (different images with similar captions).
- You want to scale batch sizes beyond softmax contrastive limits.
- You want a simpler loss function that doesn't require global synchronization.
Do not use when
- Softmax contrastive loss is already working well on your clean data.
- You need fine-grained ranking between candidates (sigmoid treats each pair independently).
- You are using very small batch sizes where softmax provides useful normalization.
Required inputs
- image_text_pairs: Image-text pairs for training.
- sigmoid_loss_config: Sigmoid loss with learnable temperature and bias parameters.
- dual_encoder: ViT + text encoder architecture.
Optional inputs
- chunked_loss_computation: Method for computing sigmoid loss over large batches without full materialization.
Outputs
- siglip_model: Dual-encoder model trained with sigmoid loss.
- improved_embeddings: Image and text embeddings with better noise robustness.
Assumptions and prerequisites
- Softmax contrastive loss creates false negatives when different images have similar captions.
- Sigmoid loss treating each pair independently avoids this problem.
- Sigmoid loss enables simpler distributed training without global batch synchronization.
Procedure
- Replace softmax with sigmoid loss Action: Change the contrastive loss from softmax (InfoNCE) to binary sigmoid on each image-text pair. Why: Sigmoid loss avoids false negative issues inherent in softmax over large batches. Note: See paper for details.
- Add learnable temperature and bias Action: Include learnable temperature and bias parameters in the sigmoid loss. Why: Adaptable parameters allow the loss to calibrate during training. Note: See paper for details.
- Implement chunked computation Action: Compute sigmoid loss in chunks to handle large batch sizes efficiently. Why: Avoids materializing the full batch similarity matrix. Note: See paper for details.
- Train on web-scale data Action: Train SigLIP on the same data as CLIP but with the new loss function. Why: Demonstrates that the loss change alone improves performance. Note: See paper for details.
- Evaluate zero-shot transfer Action: Compare SigLIP vs CLIP on zero-shot ImageNet and transfer benchmarks. Why: Validates that sigmoid loss improves over softmax contrastive loss. Note: See paper for details.
Parameters to set
- temperature — Role: Learnable temperature for sigmoid scaling. How to set: Initialize and learn during training. Default/range: Learnable. Effect: Controls the sharpness of positive/negative discrimination.
- bias — Role: Learnable bias in the sigmoid. How to set: Initialize and learn during training. Default/range: Learnable. Effect: Shifts the decision boundary for positive vs negative pairs.
- batch_size — Role: Training batch size (can be very large with sigmoid). How to set: Scale as large as GPU memory allows. Default/range: 32K+. Effect: Larger batches improve performance more with sigmoid than softmax.
Validation checks
- SigLIP should match or exceed CLIP accuracy on ImageNet zero-shot.
- Performance should scale better with batch size compared to softmax loss.
- The model should be more robust to noisy/ambiguous image-text pairs.
Failure modes
- Sigmoid loss may converge slower than softmax in the early stages.
- Very small batch sizes may not provide enough negatives for sigmoid.
- The learnable bias may need careful initialization.
Adaptation notes for VLM training
- SigLIP encoders are now preferred over CLIP encoders in many VLMs.
- The sigmoid loss principle applies to any contrastive multimodal learning.
- Use SigLIP as the vision encoder for LLaVA-style architectures.
Implementation notes
- Implement chunked sigmoid computation for memory efficiency.
- Use gradient checkpointing for very large models.
- Monitor the learned temperature and bias values during training.
Evidence from the paper
- SigLIP achieves 84.5% zero-shot ImageNet accuracy with ViT-g/14, outperforming CLIP.
- Sigmoid loss avoids false negatives from softmax normalization over large batches.
- SigLIP enables simpler distributed training by removing the need for global batch synchronization.
- The sigmoid loss scales better to very large batch sizes than softmax contrastive loss.
Source paper
- Title: Sigmoid Loss for Language Image Pre-Training
- Year: 2023
- Venue: ICCV
- Paper ID: arxiv-2303.15343v4
- URL: http://arxiv.org/abs/2303.15343v4
- arXiv ID: 2303.15343v4