Transfer Learning Patterns

Transfer Learning and Domain Adaptation

Transferring knowledge from pre-trained models to new tasks and domains — fine-tuning strategies, feature extraction, domain adaptation, and multi-task learning patterns.

When to Use

• You have a small dataset but access to a large pre-trained model
• Your target domain differs from the source domain (distribution shift)
• Training from scratch is too expensive (compute, data, or time)
• You need to adapt a general model to a specific use case
• Building multi-task systems that share representations

Transfer Learning Strategies

Feature Extraction

import torch
import torch.nn as nn
import torchvision.models as models

class FeatureExtractor:
    """Use pre-trained model as fixed feature extractor."""
    
    def __init__(self, model_name='resnet50', device='cuda'):
        # Load pre-trained model without classification head
        weights = 'IMAGENET1K_V2'
        model = getattr(models, model_name)(weights=weights)
        self.model = nn.Sequential(*list(model.children())[:-1])
        self.model.eval()
        self.model.to(device)
        self.device = device
    
    @torch.no_grad()
    def extract(self, images):
        """Extract features (no gradients)."""
        features = self.model(images)
        return features.squeeze(-1).squeeze(-1)  # Remove spatial dims


# Train a simple classifier on extracted features
def train_on_features(model, train_loader, num_classes, feature_dim=2048):
    classifier = nn.Linear(feature_dim, num_classes)
    optimizer = torch.optim.Adam(classifier.parameters(), lr=1e-3)
    criterion = nn.CrossEntropyLoss()
    
    # Pre-compute all features (fast epoch training)
    all_features, all_labels = [], []
    with torch.no_grad():
        for images, labels in train_loader:
            features = model.extract(images.to(model.device))
            all_features.append(features.cpu())
            all_labels.append(labels)
    
    all_features = torch.cat(all_features)
    all_labels = torch.cat(all_labels)
    
    for epoch in range(100):
        preds = classifier(all_features)
        loss = criterion(preds, all_labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
    return classifier

Fine-Tuning

Full Fine-Tuning

def full_finetune(model, train_loader, val_loader, num_classes, epochs=10):
    """Fine-tune the entire model."""
    # Replace classification head
    in_features = model.classifier[0].in_features
    model.classifier = nn.Linear(in_features, num_classes)
    
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)  # Lower LR
    scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, epochs)
    criterion = nn.CrossEntropyLoss()
    
    for epoch in range(epochs):
        model.train()
        for images, labels in train_loader:
            preds = model(images)
            loss = criterion(preds, labels)
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        
        scheduler.step()
        
        # Validation
        model.eval()
        val_acc = compute_accuracy(model, val_loader)
        print(f"Epoch {epoch}: val_acc={val_acc:.4f}")
    
    return model

Progressive Unfreezing

class ProgressiveUnfreeze:
    """Gradually unfreeze layers from top to bottom.
    
    Phase 1: Train new head only (epochs 1-2)
    Phase 2: Unfreeze last block (epochs 3-4)
    Phase 3: Unfreeze more blocks (epochs 5-10)
    """
    
    def __init__(self, model, num_stages=4):
        self.model = model
        self.stages = num_stages
        
        # Freeze all
        for param in model.parameters():
            param.requires_grad = False
    
    def stage(self, stage_num, lr=1e-4):
        """Configure model for training stage."""
        if stage_num == 0:
            # Only train the new head
            for param in self.model.head.parameters():
                param.requires_grad = True
        
        elif stage_num == 1:
            # Unfreeze last block
            for param in self.model.blocks[-1].parameters():
                param.requires_grad = True
        
        elif stage_num >= self.stages - 1:
            # Unfreeze everything
            for param in self.model.parameters():
                param.requires_grad = True
    
    def train(self, train_loader, val_loader, epochs_per_stage=3, base_lr=1e-4):
        for stage in range(self.stages):
            self.stage(stage, base_lr)
            lr = base_lr * (10 ** (-stage))  # Lower LR for early layers
            optimizer = torch.optim.Adam(
                [p for p in self.model.parameters() if p.requires_grad], lr=lr
            )
            
            for epoch in range(epochs_per_stage):
                self._train_epoch(optimizer, train_loader)
                acc = self._evaluate(val_loader)
                print(f"Stage {stage}, Epoch {epoch}: val_acc={acc:.4f}")

Domain Adaptation

Adversarial Domain Adaptation

class DomainAdversarialNetwork(nn.Module):
    """Domain-adversarial training (Ganin et al., 2016).
    
    Feature extractor trained to fool domain classifier,
    so features become domain-invariant."""
    
    def __init__(self, feature_extractor, num_classes):
        super().__init__()
        self.feature_extractor = feature_extractor
        self.class_classifier = nn.Linear(512, num_classes)
        self.domain_classifier = nn.Sequential(
            nn.Linear(512, 256), nn.ReLU(), nn.Linear(256, 2)  # Source vs target
        )
    
    def forward(self, x, alpha=0.1):
        features = self.feature_extractor(x)
        
        # Gradient reversal layer (implemented via gradient scaling)
        # In forward pass: identity
        # In backward pass: multiply gradients by -alpha
        
        class_output = self.class_classifier(features)
        domain_output = self.domain_classifier(
            GradientReversal.apply(features, alpha)
        )
        
        return class_output, domain_output

Common Pitfalls

1. Catastrophic forgetting — fine-tuning on new task destroys pre-trained knowledge; use low LR, shorter training
2. Domain shift — pre-trained on ImageNet, deployed on medical images; may need domain adaptation
3. Head vs body LR — new head needs higher LR than pre-trained body; use different LR groups
4. Overly aggressive fine-tuning — full fine-tuning on tiny dataset overfits; freeze more or use LoRA
5. Batch norm adaptation — batchnorm statistics need updating on new domain; use small batches or freeze BN
6. Task mismatch — pre-trained on classification, fine-tuned on detection; low-level features still transfer

Verification Checklist

• Pre-trained model's input format matches target data (size, normalization, channels)
• Feature extractor approach evaluated before full fine-tuning
• Learning rate for fine-tuning lower than from-scratch (1/10th)
• Catastrophic forgetting checked (pre-training task accuracy preserved)
• Domain adaptation considered if source/target distributions differ significantly
• Progressive unfreezing beats one-shot fine-tuning
• No overfitting on small target dataset (check val vs train gap)

See Also

• self-supervised-learning — pre-training without labels
• llm-fine-tuning-lora — efficient LLM fine-tuning
• meta-learning-few-shot — learning from very few examples
• ml-pipeline-design — integrating transfer in pipelines