adu2021 Technique: RandOpt—Ensembling via Random Perturbations of Pretrained Weights
Traditional fine-tuning assumes isolated optimal solutions in the loss landscape. Neural Thickets reveals a different regime for large pretrained models: the neighborhood around pretrained weights contains abundant diverse task-improving specialists. This "thicket regime" enables a surprisingly simple approach: randomly perturb the pretrained weights, select performers, and ensemble via majority voting.
This emerges from the fundamental difference in loss landscapes between small models ("needle in haystack" sparsity) and large pretrained models (dense solution neighborhoods).
Core Concept
RandOpt operates in two phases:
Training Phase: Generate N random Gaussian perturbations of pretrained weights θ' = θ + σ·ϵ, evaluate on validation set, select top-K performers.
Inference Phase: Generate predictions using only the K selected models, aggregate via majority voting.
The method requires no additional training—just sampling, selection, and ensemble voting. Yet it often outperforms standard fine-tuning by 5-15% on diverse tasks.
Architecture Overview
- Pretrained backbone: Frozen weight source θ
- Perturbation sampler: Generates N random Gaussian variants
- Performance evaluator: Validation-based selection of top-K
- Ensemble pool: K model copies with different weights
- Aggregator: Majority voting for final predictions
Implementation Steps
Step 1: Generate Random Weight Perturbations
Sample Gaussian noise, create perturbed model copies, and evaluate on validation set.
import torch
import numpy as np
class RandOptEnsemble:
def __init__(self, base_model, perturbation_scale=0.1, k_selected=5):
self.base_model = base_model
self.perturbation_scale = perturbation_scale
self.k_selected = k_selected
self.selected_models = []
self.base_weights = self.get_model_weights(base_model)
def get_model_weights(self, model):
"""Extract flattened weights from model."""
weights = []
for param in model.parameters():
weights.append(param.data.clone().flatten())
return torch.cat(weights)
def set_model_weights(self, model, weights):
"""Set model weights from flattened tensor."""
offset = 0
for param in model.parameters():
param_size = param.numel()
param.data = weights[offset:offset + param_size].reshape(param.shape)
offset += param_size
def generate_perturbed_models(self, num_perturbations=100):
"""
Create N random perturbations of the base model.
"""
perturbed_models = []
for _ in range(num_perturbations):
# Generate random Gaussian perturbation
perturbation = torch.randn_like(self.base_weights) * self.perturbation_scale
# Create perturbed weights
perturbed_weights = self.base_weights + perturbation
# Clone base model and set perturbed weights
perturbed_model = self.clone_model(self.base_model)
self.set_model_weights(perturbed_model, perturbed_weights)
perturbed_models.append(perturbed_model)
return perturbed_models
def clone_model(self, model):
"""Deep clone a model."""
import copy
return copy.deepcopy(model)
Step 2: Evaluate and Select Top-K Models
Benchmark perturbations on validation set, retain only the best performers.
def select_top_k_models(
perturbed_models,
validation_data,
task_metric,
k=5,
batch_size=32
):
"""
Evaluate all perturbations, return top-k by validation metric.
task_metric: function(outputs, targets) -> score
"""
model_scores = []
for idx, model in enumerate(perturbed_models):
model.eval()
total_score = 0
num_batches = 0
with torch.no_grad():
for batch_idx, (inputs, targets) in enumerate(validation_data):
if batch_idx * batch_size >= 1000: # Use subset for speed
break
outputs = model(inputs)
score = task_metric(outputs, targets)
total_score += score.item()
num_batches += 1
avg_score = total_score / num_batches
model_scores.append((idx, avg_score, model))
# Sort by performance
model_scores.sort(key=lambda x: x[1], reverse=True)
# Select top-k
selected = model_scores[:k]
return [model for _, _, model in selected]
Step 3: Ensemble Inference via Majority Voting
For classification, aggregate predictions across ensemble members.
class RandOptEnsembleInference:
def __init__(self, selected_models):
self.selected_models = selected_models
def forward(self, inputs):
"""
inputs: batch of examples
returns: ensemble predictions via majority voting
"""
batch_size = inputs.shape[0]
num_models = len(self.selected_models)
# Collect predictions from all models
all_predictions = []
for model in self.selected_models:
model.eval()
with torch.no_grad():
outputs = model(inputs)
# For classification: argmax to get class
predictions = torch.argmax(outputs, dim=-1) # (batch_size,)
all_predictions.append(predictions)
# Stack: (num_models, batch_size)
all_predictions = torch.stack(all_predictions)
# Majority voting
ensemble_predictions = torch.mode(all_predictions, dim=0)[0]
return ensemble_predictions
def forward_with_confidence(self, inputs):
"""
Also return confidence from voting agreement.
"""
batch_size = inputs.shape[0]
num_models = len(self.selected_models)
all_predictions = []
for model in self.selected_models:
model.eval()
with torch.no_grad():
outputs = model(inputs)
predictions = torch.argmax(outputs, dim=-1)
all_predictions.append(predictions)
all_predictions = torch.stack(all_predictions)
ensemble_predictions = torch.mode(all_predictions, dim=0)[0]
# Confidence: fraction voting for majority
agreement = (all_predictions == ensemble_predictions.unsqueeze(0)).float()
confidence = agreement.mean(dim=0) # (batch_size,)
return ensemble_predictions, confidence
Step 4: Multi-Task Adaptation
Extend RandOpt to multi-task scenarios with shared perturbations.
def randopt_multitask_adaptation(
base_model,
tasks,
perturbation_scale=0.1,
num_perturbations=100,
k_per_task=5
):
"""
Adapt base model to multiple tasks via perturbations.
tasks: list of (task_name, validation_data, metric_fn)
"""
# Generate shared perturbations
base_weights = torch.cat([p.data.flatten() for p in base_model.parameters()])
perturbed_models_all = []
for _ in range(num_perturbations):
perturbation = torch.randn_like(base_weights) * perturbation_scale
perturbed_weights = base_weights + perturbation
perturbed_models_all.append(perturbed_weights)
# Per-task selection
task_ensembles = {}
for task_name, val_data, metric_fn in tasks:
# Evaluate all perturbations on this task
model_scores = []
for weight_vec in perturbed_models_all:
# Create model with these weights
temp_model = create_model_with_weights(base_model, weight_vec)
# Evaluate
score = evaluate_model(temp_model, val_data, metric_fn)
model_scores.append((score, weight_vec))
# Select top-k for this task
model_scores.sort(key=lambda x: x[0], reverse=True)
top_weights = [w for _, w in model_scores[:k_per_task]]
task_ensembles[task_name] = top_weights
return task_ensembles
Step 5: Practical Integration Example
End-to-end example showing RandOpt for adaptation.
def adapt_model_with_randopt(
base_model,
validation_data,
test_data,
task_metric,
num_perturbations=100,
perturbation_scale=0.1,
k_selected=5
):
"""
Full RandOpt pipeline: perturb, select, ensemble.
"""
# Step 1: Generate perturbations
ensemble = RandOptEnsemble(base_model, perturbation_scale, k_selected)
perturbed_models = ensemble.generate_perturbed_models(num_perturbations)
# Step 2: Select top-k
selected_models = select_top_k_models(
perturbed_models,
validation_data,
task_metric,
k=k_selected
)
print(f"Selected {len(selected_models)} specialist models from {num_perturbations} perturbations")
# Step 3: Ensemble inference
ensemble_model = RandOptEnsembleInference(selected_models)
# Evaluate on test set
total_correct = 0
total_samples = 0
with torch.no_grad():
for inputs, targets in test_data:
predictions, confidence = ensemble_model.forward_with_confidence(inputs)
correct = (predictions == targets).sum().item()
total_correct += correct
total_samples += targets.shape[0]
accuracy = total_correct / total_samples
print(f"Test Accuracy: {accuracy:.4f}")
return ensemble_model, accuracy
Practical Guidance
When to Use:
- Adapting large pretrained models to new tasks
- Multi-task scenarios requiring diverse specialists
- Scenarios where ensemble voting aligns with task objectives
- Low-budget adaptation (no fine-tuning needed)
When NOT to Use:
- Single-task scenarios (use standard fine-tuning)
- Real-time inference where latency is critical (ensemble requires K forward passes)
- Tasks where diverse predictions hurt (require confidence-weighted outputs)
- Small models where "thicket regime" doesn't apply
Hyperparameter Tuning:
- perturbation_scale σ: 0.05-0.2; larger = more diversity, higher variance
- num_perturbations: 50-200; more thorough but slower
- k_selected: 3-10; balance diversity and computational cost
- validation set size: Use 10-20% of data for selection
Common Pitfalls:
- Using validation performance on held-out test set (causes overfitting to validation)
- Perturbation scale too large causing out-of-distribution behavior
- Insufficient perturbations missing good specialists
- Forgetting to freeze base model weights (no training, only sampling!)