# Self Supervised Pretraining

> Self-supervised visual representation learning. Contrastive, masked, distillation, and JEPA methods.

- Skill: `aselimc/self-supervised-pretraining` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aselimc/self-supervised-pretraining`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aselimc/self-supervised-pretraining/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: aselimc (https://skillmd.com/u/aselimc)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/aselimc/self-supervised-pretraining

---


# Self-Supervised Pretraining

## Method Families
| Family | Methods | Key Mechanism |
|--------|---------|--------------|
| Contrastive | SimCLR, MoCo v3 | Positive/negative pairs, InfoNCE loss |
| Masked | MAE, BEiT | Reconstruct masked patches |
| Distillation | DINO, DINOv2 | EMA teacher, self-distillation |
| JEPA | I-JEPA, V-JEPA | Predict representations in latent space |

## Common Components
- **EMA teacher**: `teacher = momentum * teacher + (1 - momentum) * student`
- **Multi-crop**: 2 global views (224px) + N local views (96px)
- **Masking**: random/block masking with 60-80% mask ratio (MAE-style)
- **Collapse prevention**: centering, sharpening, stop-gradient

## Evaluation Protocol
1. **Linear probe**: freeze encoder, train linear classifier
2. **kNN**: k=20 nearest neighbors on frozen features
3. **Fine-tuning**: full or partial fine-tuning on downstream
4. Report on ImageNet-1k or domain-specific benchmark

## Key Libraries
timm, torchvision, lightly