# Neural Thickets Dense Task Experts > Discover that large pretrained models have dense neighborhoods of task-specific experts—random weight perturbations improve performance. Use RandOpt: sample perturbations, select top performers, ensemble via voting for multi-task adaptation. - Skill: `adu2021/neural-thickets-dense-task-experts` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/neural-thickets-dense-task-experts` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/neural-thickets-dense-task-experts/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Productivity - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/neural-thickets-dense-task-experts --- # Technique: RandOpt—Ensembling via Random Perturbations of Pretrained Weights Traditional fine-tuning assumes isolated optimal solutions in the loss landscape. Neural Thickets reveals a different regime for large pretrained models: the neighborhood around pretrained weights contains **abundant diverse task-improving specialists**. This "thicket regime" enables a surprisingly simple approach: randomly perturb the pretrained weights, select performers, and ensemble via majority voting. This emerges from the fundamental difference in loss landscapes between small models ("needle in haystack" sparsity) and large pretrained models (dense solution neighborhoods). ## Core Concept RandOpt operates in two phases: **Training Phase**: Generate N random Gaussian perturbations of pretrained weights θ' = θ + σ·ϵ, evaluate on validation set, select top-K performers. **Inference Phase**: Generate predictions using only the K selected models, aggregate via majority voting. The method requires no additional training—just sampling, selection, and ensemble voting. Yet it often outperforms standard fine-tuning by 5-15% on diverse tasks. ## Architecture Overview - **Pretrained backbone**: Frozen weight source θ - **Perturbation sampler**: Generates N random Gaussian variants - **Performance evaluator**: Validation-based selection of top-K - **Ensemble pool**: K model copies with different weights - **Aggregator**: Majority voting for final predictions ## Implementation Steps ### Step 1: Generate Random Weight Perturbations Sample Gaussian noise, create perturbed model copies, and evaluate on validation set. ```python import torch import numpy as np class RandOptEnsemble: def __init__(self, base_model, perturbation_scale=0.1, k_selected=5): self.base_model = base_model self.perturbation_scale = perturbation_scale self.k_selected = k_selected self.selected_models = [] self.base_weights = self.get_model_weights(base_model) def get_model_weights(self, model): """Extract flattened weights from model.""" weights = [] for param in model.parameters(): weights.append(param.data.clone().flatten()) return torch.cat(weights) def set_model_weights(self, model, weights): """Set model weights from flattened tensor.""" offset = 0 for param in model.parameters(): param_size = param.numel() param.data = weights[offset:offset + param_size].reshape(param.shape) offset += param_size def generate_perturbed_models(self, num_perturbations=100): """ Create N random perturbations of the base model. """ perturbed_models = [] for _ in range(num_perturbations): # Generate random Gaussian perturbation perturbation = torch.randn_like(self.base_weights) * self.perturbation_scale # Create perturbed weights perturbed_weights = self.base_weights + perturbation # Clone base model and set perturbed weights perturbed_model = self.clone_model(self.base_model) self.set_model_weights(perturbed_model, perturbed_weights) perturbed_models.append(perturbed_model) return perturbed_models def clone_model(self, model): """Deep clone a model.""" import copy return copy.deepcopy(model) ``` ### Step 2: Evaluate and Select Top-K Models Benchmark perturbations on validation set, retain only the best performers. ```python def select_top_k_models( perturbed_models, validation_data, task_metric, k=5, batch_size=32 ): """ Evaluate all perturbations, return top-k by validation metric. task_metric: function(outputs, targets) -> score """ model_scores = [] for idx, model in enumerate(perturbed_models): model.eval() total_score = 0 num_batches = 0 with torch.no_grad(): for batch_idx, (inputs, targets) in enumerate(validation_data): if batch_idx * batch_size >= 1000: # Use subset for speed break outputs = model(inputs) score = task_metric(outputs, targets) total_score += score.item() num_batches += 1 avg_score = total_score / num_batches model_scores.append((idx, avg_score, model)) # Sort by performance model_scores.sort(key=lambda x: x[1], reverse=True) # Select top-k selected = model_scores[:k] return [model for _, _, model in selected] ``` ### Step 3: Ensemble Inference via Majority Voting For classification, aggregate predictions across ensemble members. ```python class RandOptEnsembleInference: def __init__(self, selected_models): self.selected_models = selected_models def forward(self, inputs): """ inputs: batch of examples returns: ensemble predictions via majority voting """ batch_size = inputs.shape[0] num_models = len(self.selected_models) # Collect predictions from all models all_predictions = [] for model in self.selected_models: model.eval() with torch.no_grad(): outputs = model(inputs) # For classification: argmax to get class predictions = torch.argmax(outputs, dim=-1) # (batch_size,) all_predictions.append(predictions) # Stack: (num_models, batch_size) all_predictions = torch.stack(all_predictions) # Majority voting ensemble_predictions = torch.mode(all_predictions, dim=0)[0] return ensemble_predictions def forward_with_confidence(self, inputs): """ Also return confidence from voting agreement. """ batch_size = inputs.shape[0] num_models = len(self.selected_models) all_predictions = [] for model in self.selected_models: model.eval() with torch.no_grad(): outputs = model(inputs) predictions = torch.argmax(outputs, dim=-1) all_predictions.append(predictions) all_predictions = torch.stack(all_predictions) ensemble_predictions = torch.mode(all_predictions, dim=0)[0] # Confidence: fraction voting for majority agreement = (all_predictions == ensemble_predictions.unsqueeze(0)).float() confidence = agreement.mean(dim=0) # (batch_size,) return ensemble_predictions, confidence ``` ### Step 4: Multi-Task Adaptation Extend RandOpt to multi-task scenarios with shared perturbations. ```python def randopt_multitask_adaptation( base_model, tasks, perturbation_scale=0.1, num_perturbations=100, k_per_task=5 ): """ Adapt base model to multiple tasks via perturbations. tasks: list of (task_name, validation_data, metric_fn) """ # Generate shared perturbations base_weights = torch.cat([p.data.flatten() for p in base_model.parameters()]) perturbed_models_all = [] for _ in range(num_perturbations): perturbation = torch.randn_like(base_weights) * perturbation_scale perturbed_weights = base_weights + perturbation perturbed_models_all.append(perturbed_weights) # Per-task selection task_ensembles = {} for task_name, val_data, metric_fn in tasks: # Evaluate all perturbations on this task model_scores = [] for weight_vec in perturbed_models_all: # Create model with these weights temp_model = create_model_with_weights(base_model, weight_vec) # Evaluate score = evaluate_model(temp_model, val_data, metric_fn) model_scores.append((score, weight_vec)) # Select top-k for this task model_scores.sort(key=lambda x: x[0], reverse=True) top_weights = [w for _, w in model_scores[:k_per_task]] task_ensembles[task_name] = top_weights return task_ensembles ``` ### Step 5: Practical Integration Example End-to-end example showing RandOpt for adaptation. ```python def adapt_model_with_randopt( base_model, validation_data, test_data, task_metric, num_perturbations=100, perturbation_scale=0.1, k_selected=5 ): """ Full RandOpt pipeline: perturb, select, ensemble. """ # Step 1: Generate perturbations ensemble = RandOptEnsemble(base_model, perturbation_scale, k_selected) perturbed_models = ensemble.generate_perturbed_models(num_perturbations) # Step 2: Select top-k selected_models = select_top_k_models( perturbed_models, validation_data, task_metric, k=k_selected ) print(f"Selected {len(selected_models)} specialist models from {num_perturbations} perturbations") # Step 3: Ensemble inference ensemble_model = RandOptEnsembleInference(selected_models) # Evaluate on test set total_correct = 0 total_samples = 0 with torch.no_grad(): for inputs, targets in test_data: predictions, confidence = ensemble_model.forward_with_confidence(inputs) correct = (predictions == targets).sum().item() total_correct += correct total_samples += targets.shape[0] accuracy = total_correct / total_samples print(f"Test Accuracy: {accuracy:.4f}") return ensemble_model, accuracy ``` ## Practical Guidance **When to Use:** - Adapting large pretrained models to new tasks - Multi-task scenarios requiring diverse specialists - Scenarios where ensemble voting aligns with task objectives - Low-budget adaptation (no fine-tuning needed) **When NOT to Use:** - Single-task scenarios (use standard fine-tuning) - Real-time inference where latency is critical (ensemble requires K forward passes) - Tasks where diverse predictions hurt (require confidence-weighted outputs) - Small models where "thicket regime" doesn't apply **Hyperparameter Tuning:** - **perturbation_scale σ**: 0.05-0.2; larger = more diversity, higher variance - **num_perturbations**: 50-200; more thorough but slower - **k_selected**: 3-10; balance diversity and computational cost - **validation set size**: Use 10-20% of data for selection **Common Pitfalls:** - Using validation performance on held-out test set (causes overfitting to validation) - Perturbation scale too large causing out-of-distribution behavior - Insufficient perturbations missing good specialists - Forgetting to freeze base model weights (no training, only sampling!) ## Reference [Neural Thickets paper on arXiv](https://arxiv.org/abs/2603.12228)