Scale Evolution Strategies to billion-parameter LLMs without backpropagation for superior robustness and stability across diverse models, reward horizons, and evaluation tasks. Outperforms RL methods while eliminating gradient computation overhead.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Scale Evolution Strategies to billion-parameter LLMs without backpropagation for superior robustness and stability across diverse models, reward horizons, and evaluation tasks. Outperforms RL methods while eliminating gradient computation overhead.
Evolution Strategies Fine-Tuning: Direct Parameter Optimization at Billion Scale
Outcome
Fine-tune large language models through population-based direct parameter search, achieving robust model improvements across diverse architectures with 15.5× lower training variance than gradient-based RL methods and resistance to reward hacking without explicit penalties.
Problem Context
Current LLM fine-tuning relies on backpropagation through gradient-based reinforcement learning (PPO, GRPO), which struggles with:
Sparse, long-horizon rewards: Intermediate supervision often unavailable for reasoning tasks; gradients through long sequences become unstable
Cross-model brittleness: Fine-tuning success varies dramatically across base model architectures; GRPO failed entirely on certain models
Training instability: High variance across runs (15.5× higher than ES) makes expensive fine-tuning unreliable for large deployments
Computational overhead: Backpropagation and KL penalty computation add substantial memory and compute burden
Evolution Strategies offer an alternative: direct parameter space search using only reward signals, no gradients required.
Core Concept
Evolution Strategies treat model parameters as a genome subject to evolutionary pressure. The algorithm repeatedly:
Sample parameter perturbations from a normal distribution
Evaluate perturbed models on the target task to obtain rewards
Update parameters in the direction of high-reward perturbations (natural gradient)
Key insight: ES needs only reward values, not gradients, enabling response-level supervision (did the model solve the problem?) rather than loss gradients. This decouples optimization from model architecture and enables effective search in sparse reward regimes.
At billion-parameter scale, seven engineering optimizations make ES tractable: noise reproducibility via random seeds, parallel GPU evaluation, in-place perturbation, reward normalization, greedy decoding, decomposed updates, and simplified learning rates.
Architecture Overview
Population-Based Search
Small fixed population (30 members vs. 10,000+ in prior work) evaluates perturbations in parallel
Each member: base weights + scaled Gaussian noise sampled from seed
Parallel evaluation across GPUs; single machines or distributed clusters via Hugging Face Accelerate
Reward-Driven Parameter Updates
Collect reward signal (scalar, delayed OK) from each population member
Normalize rewards to zero-mean unit-variance
Compute utility-weighted average of perturbations: Δθ ∝ Σ(utility_i × noise_i)
Apply learning rate: θ_new = θ_old + α × Δθ
Memory & Compute Efficiency
Noise retrieval: reconstruct perturbations from random seeds on-the-fly (no storage overhead)
The reward function takes a model and returns a scalar score. ES optimizes this directly—no gradients needed.
defcompute_reward(model, tokenizer, examples):
"""
Evaluate model on a task and return scalar reward.
Args:
model: LLM instance (already loaded)
tokenizer: Tokenizer for the model
examples: List of {input, expected_output} dicts
Returns:
float: Aggregated reward (0-1 range recommended)
"""
correct = 0for example in examples:
# Generate response with greedy decoding
inputs = tokenizer(example["input"], return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=256,
do_sample=False, # greedy
pad_token_id=tokenizer.eos_token_id
)
response = tokenizer.decode(output[0], skip_special_tokens=True)
# Check correctness (task-specific)if is_correct(response, example["expected_output"]):
correct += 1# Return fraction correctreturn correct / len(examples)
defis_correct(response, expected):
"""Task-specific correctness check."""# Example: exact matchreturn response.strip() == expected.strip()
3. Initialize Population and State
Set up the ES state: mean parameters, step size, and population utilities.
import torch
import numpy as np
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model_name = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Flatten parameters into a single vector (for ES state)
params_init = torch.nn.utils.parameters_to_vector(model.parameters()).detach().clone()
num_params = params_init.numel()
# ES hyperparameters
population_size = 30# Small population due to engineering optimizations
learning_rate = 0.001
sigma = 0.017# Standard deviation of perturbations (tune per task)print(f"Model parameters: {num_params:,} | Population: {population_size}")
# Initialize utilities (per-member weighting)
utilities = np.array([max(0, np.log(population_size/2 + 1) - np.log(i+1))
for i inrange(population_size)])
utilities /= np.sum(utilities) # Normalize
4. Main ES Loop: Mutation, Evaluation, and Update
Run ES for multiple generations, accumulating rewards and updating parameters.
defes_train_loop(
model, tokenizer, params_init, reward_fn,
generations=100, population_size=30, sigma=0.017, lr=0.001,
seed_base=42, device="cuda"):
"""
Main Evolution Strategies training loop.
Args:
model: LLM to fine-tune
tokenizer: Model tokenizer
params_init: Initial parameter vector
reward_fn: Function(model, tokenizer) -> float
generations: Number of ES iterations
population_size: Population members per iteration
sigma: Perturbation std dev (controls exploration)
lr: Natural gradient step size
seed_base: RNG seed for reproducibility
device: "cuda" or "cpu"
"""
params_current = params_init.clone()
rewards_history = []
for gen inrange(generations):
gen_rewards = []
param_updates = np.zeros(params_init.numel())
# Generate and evaluate populationfor member_id inrange(population_size):
# Deterministic noise from seed (no storage overhead)
seed = seed_base + gen * population_size + member_id
np.random.seed(seed)
noise = torch.tensor(
np.random.randn(params_init.numel()),
dtype=params_init.dtype,
device=device
)
# Perturbed parameters
params_perturbed = params_current + sigma * noise
# Update model weights in-place (layer by layer)
offset = 0for param in model.parameters():
param_size = param.numel()
param.data = params_perturbed[offset:offset+param_size].reshape(param.shape)
offset += param_size
# Evaluate (reward only, no gradients)
reward = reward_fn(model, tokenizer)
gen_rewards.append(reward)
# Accumulate utility-weighted noise for update
param_updates += utilities[member_id] * noise.cpu().numpy()
# Normalize rewards and update parameters
rewards_array = np.array(gen_rewards)
rewards_normalized = (rewards_array - np.mean(rewards_array)) / (np.std(rewards_array) + 1e-8)
# Natural gradient update: θ ← θ + α * (1/σ) * Σ util_i * noise_i * (r_i - mean_r)
param_updates_weighted = np.zeros_like(param_updates)
for member_id inrange(population_size):
seed = seed_base + gen * population_size + member_id
np.random.seed(seed)
noise_update = np.random.randn(params_init.numel())
param_updates_weighted += utilities[member_id] * noise_update * rewards_normalized[member_id]
params_current = params_current.cpu() + (lr / sigma) * torch.tensor(param_updates_weighted, dtype=params_current.dtype)
params_current = params_current.to(device)
# Log progress
best_reward = np.max(gen_rewards)
mean_reward = np.mean(gen_rewards)
rewards_history.append(best_reward)
if (gen + 1) % 10 == 0:
print(f"Gen {gen+1:3d} | Best: {best_reward:.4f} | Mean: {mean_reward:.4f} | Std: {np.std(gen_rewards):.4f}")
return params_current, rewards_history
5. Save and Evaluate Fine-Tuned Model
After training, restore final parameters and test performance.
defsave_finetuned_model(model, params_final, output_path):
"""
Write final parameters back to model and save to disk.
Args:
model: LLM with architecture to save
params_final: Final parameter vector from ES
output_path: Directory to save (will create via model.save_pretrained)
"""# Restore final parameters
offset = 0for param in model.parameters():
param_size = param.numel()
param.data = params_final[offset:offset+param_size].reshape(param.shape)
offset += param_size
# Save to disk
model.save_pretrained(output_path)
print(f"Fine-tuned model saved to {output_path}")
# Example usageif __name__ == "__main__":
# Load data (example: math reasoning)
train_examples = [
{"input": "Solve: 2x + 3 = 7", "expected_output": "x = 2"},
# ... more examples
]
# Define reward functiondefreward_fn(m, t):
return compute_reward(m, t, train_examples[:20]) # Subset for speed# Run ES fine-tuning
params_final, history = es_train_loop(
model, tokenizer, params_init, reward_fn,
generations=100,
population_size=30,
sigma=0.017,
lr=0.001
)
# Save and evaluate
save_finetuned_model(model, params_final, "./model_finetuned")
6. Distributed Multi-GPU Setup (via Accelerate)
For large models, distribute population evaluation across multiple GPUs or machines.
from accelerate import Accelerator
defes_train_distributed(
model_name, reward_fn,
generations=100, population_size=30,
num_processes=2, gpu_threads=15):
"""
Multi-GPU ES training using Hugging Face Accelerate.
Total parallel evaluations = num_processes * gpu_threads.
Args:
model_name: HuggingFace model ID
reward_fn: Reward function (called per process)
num_processes: Number of GPUs (or machines)
gpu_threads: Threads per GPU (model copies per GPU)
"""
accelerator = Accelerator()
# Each process loads model independently
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = accelerator.prepare(model)
# Each process evaluates a subset of population
local_pop_size = population_size // num_processes
# Main ES loop (same as single-GPU, but rewards aggregated)# ...print(f"Rank {accelerator.process_index}: evaluating {local_pop_size} members")
Practical Guidance
Hyperparameter Recommendations
Parameter
Typical Range
Notes
population_size
20–50
Smaller than RL batch sizes; 30 is default. Increase for harder tasks.
sigma (noise std)
0.01–0.05
Controls exploration vs. exploitation. Start at 0.017; lower for final refinement.
learning_rate
0.0001–0.01
Step size for parameter updates. 0.001 is standard; reduce if oscillating.
generations
50–500
Task-dependent; monitor reward curve to detect plateau.
seed_base
any
Ensures reproducibility; increment per run if multiple trials needed.
When to Use ES Fine-Tuning
Reasoning tasks with sparse, delayed rewards (math, logic, puzzle solving)
Heterogeneous base models: Need a method that works across Qwen, Llama, Mistral, etc.
Robustness critical: Training stability matters more than marginal reward gains
Reward specification difficult: You have outcome labels but not intermediate supervision
Small datasets: ES is sample-efficient (often < 20% of RL data needed)
Long-horizon tasks: Few intermediate steps; only final answer is evaluable
When NOT to Use ES Fine-Tuning
Dense reward signals: If you have loss gradients or detailed intermediate supervision, gradient-based RL (PPO, DPO) will be faster
Continuous action spaces: ES excels at large discrete parameter spaces; for action fine-tuning, RL is more direct
Extreme speed required: ES requires multiple forward passes per update; if latency is critical, SFT or single-pass methods preferred
Highly model-specific optimization: If you're tuning for a single model and have unlimited compute for gradient tuning, RL may squeeze out extra performance
Limited evaluation budget: Each generation requires population_size full model evaluations; if evaluation is expensive (e.g., human-in-the-loop), use smaller populations or RL with importance weighting
Common Pitfalls
Σ too high or low: If noise is too large, updates become random. If too small, stuck in local optima. Adapt σ per task (start 0.017, halve if rewards plateau).
Ignoring reward scale: Normalizing rewards per generation is critical for stable updates. If rewards are 0–1 vs. 0–1000, learning rate must adjust; the algorithm handles this via z-score normalization.
Small population on large tasks: With population_size < 15, gradient estimates become noisy. For complex reasoning, use 30+.
Not greedy decoding: ES assumes deterministic reward (same input → same output). Sampling during generation adds noise; use greedy decoding or fix seed.
Starting from mid-training checkpoint: ES searches from the current parameter point; if base model is undertrained, ES may optimize for weak behaviors. Fine-tune strong base models.
Incorrect utility weights: The utility vector ranks population members by reward. Ensure it's recalculated per generation (don't reuse across different tasks).
Reference
Paper: Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
Authors: Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Elliot Meyerson, Babak Hodjat, Risto Miikkulainen
ArXiv: 2509.24372Code: GitHub – Cognizant AI Lab
Cited Baselines: PPO (Schulman et al., 2017), GRPO (Xu et al., 2024), DPO (Rafailov et al., 2023)
