Discover that large pretrained models have dense neighborhoods of task-specific experts—random weight perturbations improve performance. Use RandOpt: sample perturbations, select top performers, ensemble via voting for multi-task adaptation.
Discover that large pretrained models have dense neighborhoods of task-specific experts—random weight perturbations improve performance. Use RandOpt: sample perturbations, select top performers, ensemble via voting for multi-task adaptation.
Technique: RandOpt—Ensembling via Random Perturbations of Pretrained Weights
Traditional fine-tuning assumes isolated optimal solutions in the loss landscape. Neural Thickets reveals a different regime for large pretrained models: the neighborhood around pretrained weights contains abundant diverse task-improving specialists. This "thicket regime" enables a surprisingly simple approach: randomly perturb the pretrained weights, select performers, and ensemble via majority voting.
This emerges from the fundamental difference in loss landscapes between small models ("needle in haystack" sparsity) and large pretrained models (dense solution neighborhoods).
Core Concept
RandOpt operates in two phases:
Training Phase: Generate N random Gaussian perturbations of pretrained weights θ' = θ + σ·ϵ, evaluate on validation set, select top-K performers.
Inference Phase: Generate predictions using only the K selected models, aggregate via majority voting.
The method requires no additional training—just sampling, selection, and ensemble voting. Yet it often outperforms standard fine-tuning by 5-15% on diverse tasks.
Architecture Overview
Pretrained backbone: Frozen weight source θ
Perturbation sampler: Generates N random Gaussian variants
Performance evaluator: Validation-based selection of top-K
Ensemble pool: K model copies with different weights
Aggregator: Majority voting for final predictions
Implementation Steps
Step 1: Generate Random Weight Perturbations
Sample Gaussian noise, create perturbed model copies, and evaluate on validation set.
"""
Create N random perturbations of the base model.
"""
for
in
range
# Generate random Gaussian perturbation
self
self
# Create perturbed weights
self
# Clone base model and set perturbed weights
self
self
self
return
def
clone_model
self, model
"""Deep clone a model."""
import
return
Step 2: Evaluate and Select Top-K Models
Benchmark perturbations on validation set, retain only the best performers.
defselect_top_k_models(
perturbed_models,
validation_data,
task_metric,
k=5,
batch_size=32):
"""
Evaluate all perturbations, return top-k by validation metric.
task_metric: function(outputs, targets) -> score
"""
model_scores = []
for idx, model inenumerate(perturbed_models):
model.eval()
total_score = 0
num_batches = 0with torch.no_grad():
for batch_idx, (inputs, targets) inenumerate(validation_data):
if batch_idx * batch_size >= 1000: # Use subset for speedbreak
outputs = model(inputs)
score = task_metric(outputs, targets)
total_score += score.item()
num_batches += 1
avg_score = total_score / num_batches
model_scores.append((idx, avg_score, model))
# Sort by performance
model_scores.sort(key=lambda x: x[1], reverse=True)
# Select top-k
selected = model_scores[:k]
return [model for _, _, model in selected]
Step 3: Ensemble Inference via Majority Voting
For classification, aggregate predictions across ensemble members.
classRandOptEnsembleInference:
def__init__(self, selected_models):
self.selected_models = selected_models
defforward(self, inputs):
"""
inputs: batch of examples
returns: ensemble predictions via majority voting
"""
batch_size = inputs.shape[0]
num_models = len(self.selected_models)
# Collect predictions from all models
all_predictions = []
for model inself.selected_models:
model.eval()
with torch.no_grad():
outputs = model(inputs)
# For classification: argmax to get class
predictions = torch.argmax(outputs, dim=-1) # (batch_size,)
all_predictions.append(predictions)
# Stack: (num_models, batch_size)
all_predictions = torch.stack(all_predictions)
# Majority voting
ensemble_predictions = torch.mode(all_predictions, dim=0)[0]
return ensemble_predictions
defforward_with_confidence(self, inputs):
"""
Also return confidence from voting agreement.
"""
batch_size = inputs.shape[0]
num_models = len(self.selected_models)
all_predictions = []
for model inself.selected_models:
model.eval()
with torch.no_grad():
outputs = model(inputs)
predictions = torch.argmax(outputs, dim=-1)
all_predictions.append(predictions)
all_predictions = torch.stack(all_predictions)
ensemble_predictions = torch.mode(all_predictions, dim=0)[0]
# Confidence: fraction voting for majority
agreement = (all_predictions == ensemble_predictions.unsqueeze(0)).float()
confidence = agreement.mean(dim=0) # (batch_size,)return ensemble_predictions, confidence
Step 4: Multi-Task Adaptation
Extend RandOpt to multi-task scenarios with shared perturbations.
defrandopt_multitask_adaptation(
base_model,
tasks,
perturbation_scale=0.1,
num_perturbations=100,
k_per_task=5):
"""
Adapt base model to multiple tasks via perturbations.
tasks: list of (task_name, validation_data, metric_fn)
"""# Generate shared perturbations
base_weights = torch.cat([p.data.flatten() for p in base_model.parameters()])
perturbed_models_all = []
for _ inrange(num_perturbations):
perturbation = torch.randn_like(base_weights) * perturbation_scale
perturbed_weights = base_weights + perturbation
perturbed_models_all.append(perturbed_weights)
# Per-task selection
task_ensembles = {}
for task_name, val_data, metric_fn in tasks:
# Evaluate all perturbations on this task
model_scores = []
for weight_vec in perturbed_models_all:
# Create model with these weights
temp_model = create_model_with_weights(base_model, weight_vec)
# Evaluate
score = evaluate_model(temp_model, val_data, metric_fn)
model_scores.append((score, weight_vec))
# Select top-k for this task
model_scores.sort(key=lambda x: x[0], reverse=True)
top_weights = [w for _, w in model_scores[:k_per_task]]
task_ensembles[task_name] = top_weights
return task_ensembles
Step 5: Practical Integration Example
End-to-end example showing RandOpt for adaptation.
