# Self Supervised Learning Advanced

> Use when implementing advanced self-supervised learning.

- Skill: `loopyluci/self-supervised-learning-advanced` (Agent Skill)
- Install (CLI): `npx skillmds@latest add loopyluci/self-supervised-learning-advanced`
- Raw SKILL.md: https://api.skillmd.com/api/skills/loopyluci/self-supervised-learning-advanced/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: LoopyLuci (https://skillmd.com/u/loopyluci)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/loopyluci/self-supervised-learning-advanced

---


# Advanced Self-Supervised Learning

Implementing advanced self-supervised learning — from contrastive methods (SimCLR, MoCo, BYOL) through masked modeling (MAE, BERT-style) and joint embedding architectures.

## When to Use

- Pre-training models without labeled data
- Learning representations from unlabeled data
- Improving downstream task performance with SSL pre-training
- Reducing labeled data requirements for fine-tuning

## SSL Methods

```python
SSL_METHODS = {
    'contrastive': 'SimCLR, MoCo — pull positive pairs together, push negatives apart',
    'masked': 'MAE, BERT — mask input patches/tokens, reconstruct missing content',
    'clustering': 'SwAV, DeepCluster — cluster representations, enforce cluster consistency',
    'distillation': 'BYOL, DINO — student-teacher with no negatives, momentum encoder',
    'whitening': 'W-MSE, Barlow Twins — decorrelate feature dimensions, redundancy reduction',
}

class SimCLR:
    """Simplified SimCLR contrastive learning."""
    def __init__(self, encoder, projection_dim=128, temperature=0.5):
        self.encoder = encoder
        self.projection = nn.Linear(encoder.output_dim, projection_dim)
        self.temperature = temperature
    
    def contrastive_loss(self, z_i, z_j):
        """NT-Xent loss — InfoNCE with temperature."""
        z_i = F.normalize(z_i, dim=1)
        z_j = F.normalize(z_j, dim=1)
        representations = torch.cat([z_i, z_j], dim=0)
        similarity = representations @ representations.T / self.temperature
        # Labels: diagonal = positive pairs
        batch_size = z_i.shape[0]
        labels = torch.arange(batch_size, device=z_i.device)
        labels = torch.cat([labels + batch_size, labels])
        mask = ~torch.eye(2 * batch_size, dtype=bool)
        return F.cross_entropy(similarity[mask].view(2 * batch_size, -1), labels)
```

## Verification Checklist

- [ ] SSL method matched to data type (contrastive for vision, masked for text)
- [ ] Strong augmentation pipeline for positive pair generation
- [ ] Projection head used for representation learning
- [ ] Training stabilizes without collapse (monitor uniformity + alignment)
- [ ] Downstream task improvement measured (with vs without SSL pre-training)
- [ ] Batch size sufficient for contrastive methods (256+ recommended)