Self Supervised Learning Advanced

Advanced Self-Supervised Learning

Implementing advanced self-supervised learning — from contrastive methods (SimCLR, MoCo, BYOL) through masked modeling (MAE, BERT-style) and joint embedding architectures.

When to Use

• Pre-training models without labeled data
• Learning representations from unlabeled data
• Improving downstream task performance with SSL pre-training
• Reducing labeled data requirements for fine-tuning

SSL Methods

SSL_METHODS = {
    'contrastive': 'SimCLR, MoCo — pull positive pairs together, push negatives apart',
    'masked': 'MAE, BERT — mask input patches/tokens, reconstruct missing content',
    'clustering': 'SwAV, DeepCluster — cluster representations, enforce cluster consistency',
    'distillation': 'BYOL, DINO — student-teacher with no negatives, momentum encoder',
    'whitening': 'W-MSE, Barlow Twins — decorrelate feature dimensions, redundancy reduction',
}

class SimCLR:
    """Simplified SimCLR contrastive learning."""
    def __init__(self, encoder, projection_dim=128, temperature=0.5):
        self.encoder = encoder
        self.projection = nn.Linear(encoder.output_dim, projection_dim)
        self.temperature = temperature
    
    def contrastive_loss(self, z_i, z_j):
        """NT-Xent loss — InfoNCE with temperature."""
        z_i = F.normalize(z_i, dim=1)
        z_j = F.normalize(z_j, dim=1)
        representations = torch.cat([z_i, z_j], dim=0)
        similarity = representations @ representations.T / self.temperature
        # Labels: diagonal = positive pairs
        batch_size = z_i.shape[0]
        labels = torch.arange(batch_size, device=z_i.device)
        labels = torch.cat([labels + batch_size, labels])
        mask = ~torch.eye(2 * batch_size, dtype=bool)
        return F.cross_entropy(similarity[mask].view(2 * batch_size, -1), labels)

Verification Checklist

• SSL method matched to data type (contrastive for vision, masked for text)
• Strong augmentation pipeline for positive pair generation
• Projection head used for representation learning
• Training stabilizes without collapse (monitor uniformity + alignment)
• Downstream task improvement measured (with vs without SSL pre-training)
• Batch size sufficient for contrastive methods (256+ recommended)