Skip to main content Startseite Ersteller adu2021 skillxiv quantized-evolution-strategies
quantized-evolution-strategies Fine-tune quantized LLMs directly in low-precision discrete parameter space using evolution strategies with accumulated error feedback. Overcome gradient stagnation in quantized models by accumulating fractional updates using Delta-Sigma modulation, achieving significant improvements in INT4 quantized models without full-precision gradients.
Zur Installation springen Skills Marktplatz Entdecken und erkunden Sie KI-Skills, die von der Community erstellt wurden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Prompt-Details anzeigen Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
npx skills add https://github.com/ADu2021/skillXiv --skill quantized-evolution-strategiesDer Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
ZIP herunterladen Herunterladen... Mehr aus diesem Repository meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Verwandte Berufe SOC
Basierend auf der SOC-Berufsklassifikation
name quantized-evolution-strategies title Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2602.03120 keywords ["Quantization","Fine-Tuning","Evolution Strategies","Gradient Stagnation","Memory Efficiency"] description Fine-tune quantized LLMs directly in low-precision discrete parameter space using evolution strategies with accumulated error feedback. Overcome gradient stagnation in quantized models by accumulating fractional updates using Delta-Sigma modulation, achieving significant improvements in INT4 quantized models without full-precision gradients.
Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs
Problem Context
Quantized language models (INT4, INT8) reduce memory but create a fundamental optimization problem: gradient updates often fall below quantization thresholds, causing training to stagnate. Standard backprop doesn't work because discrete rounding eliminates infinitesimal signals. Quantized Evolution Strategies (QES) solves this using population-based perturbations and accumulated error feedback inspired by signal processing.
Core Concept
QES operates in three phases: (1) estimate gradients using population-based perturbations without backprop, (2) accumulate error residuals at each iteration (Delta-Sigma modulation), (3) make discrete updates only when accumulated error crosses rounding threshold. This transforms infinitesimal signals into discrete changes within quantized parameter space.
Architecture Overview
Population-based gradients : Estimate gradients via finite differences on random perturbationsError accumulation : Track fractional updates that don't cross quantization boundariesDelta-Sigma feedback : Accumulated residuals trigger discrete parameter changesStateless replay : Reconstruct optimization history from seeds, not full stateDirect quantization optimization : No full-precision shadow parameters
Implementation
Step 1: Population-based gradient estimation
import torch
import numpy as np
from typing import Tuple
class PopulationGradientEstimator :
"""Estimate gradients from function evaluations (no backprop)."""
def __init__ (
self,
pop_size: int = 32 ,
sigma: float = 0.01 ,
learning_rate: float = 1e-3
):
self .pop_size = pop_size
self .sigma = sigma
self
def
estimate_gradients
self,
model: torch.nn.Module,
reward_fn,
batch: dict
dict
"""
Estimate gradients using population sampling.
Args:
model: Quantized language model
reward_fn: Function returning scalar reward
batch: Input batch
Returns:
estimated_grads: Per-parameter gradient estimates
"""
for
in
list
for
in
range
self
for
in
self
with
for
in
with
for
in
for
in
for
in
range
self
self
self
2
return
Step 2: Accumulated error feedback (Delta-Sigma modulation) class AccumulatedErrorFeedback :
"""Accumulate fractional updates until crossing quantization threshold."""
def __init__ (self, quantization_bits: int = 4 ):
self .quantization_bits = quantization_bits
self .quantization_step = 1.0 / (2 ** quantization_bits)
self .error_buffer = {}
def apply_accumulated_update (
self,
param_tensor: torch.Tensor,
param_name: str ,
gradient: torch.Tensor,
learning_rate: float
) -> Tuple [torch.Tensor, float ]:
"""
Apply update with error accumulation.
Strategy:
- Compute update: delta = lr * gradient
- Add to error buffer: error[name] += delta
- When |error| > quantization_step: make discrete update
"""
if param_name not in self .error_buffer:
self .error_buffer[param_name] = torch.zeros_like(param_tensor)
proposed_update = learning_rate * gradient
self .error_buffer[param_name].add_(proposed_update)
error = self .error_buffer[param_name]
discrete_update = torch.sign(error) * self .quantization_step
crosses_threshold = torch.abs (error) >= self .quantization_step
actual_update = discrete_update.clone()
actual_update[~crosses_threshold] = 0
param_tensor.add_(actual_update)
self .error_buffer[param_name][crosses_threshold] = 0
num_updates = crosses_threshold.sum ().item()
return param_tensor, num_updates
Step 3: Stateless seed replay optimization class StatelessSeedReplay :
"""Reconstruct optimization state from random seeds without memory overhead."""
def __init__ (self ):
self .seed_history = []
self .reward_history = []
def save_iteration_state (self, seed: int , reward: float ):
"""Save only seed and reward, not full gradient state."""
self .seed_history.append(seed)
self .reward_history.append(reward)
def reconstruct_gradients (
self,
model: torch.nn.Module,
iteration: int ,
sigma: float = 0.01
) -> dict :
"""
Reconstruct gradients at specific iteration from saved seeds.
Avoids storing full gradient history.
"""
torch.manual_seed(self .seed_history[iteration])
params = {name: p for name, p in model.named_parameters()}
estimated_grads = {}
baseline_reward = self .reward_history[iteration]
for i in range (32 ):
pert_dict = {}
for name in params.keys():
pert = sigma * torch.randn_like(params[name])
pert_dict[name] = pert
if i < len (self .reward_history):
reward = self .reward_history[i + iteration * 32 ]
else :
reward = baseline_reward
for name, pert in pert_dict.items():
advantage = reward - baseline_reward
if name not in estimated_grads:
estimated_grads[name] = torch.zeros_like(params[name])
estimated_grads[name].add_(advantage * pert)
for name in estimated_grads:
estimated_grads[name].div_(32 * sigma ** 2 )
return estimated_grads
Step 4: QES training step class QuantizedEvolutionStrategies :
"""Full QES optimizer for quantized model fine-tuning."""
def __init__ (
self,
model: torch.nn.Module,
reward_fn,
quantization_bits: int = 4 ,
pop_size: int = 32 ,
learning_rate: float = 1e-3 ,
use_seed_replay: bool = True
):
self .model = model
self .reward_fn = reward_fn
self .pop_size = pop_size
self .learning_rate = learning_rate
self .grad_estimator = PopulationGradientEstimator(
pop_size=pop_size, sigma=0.01 , learning_rate=learning_rate
)
self .error_feedback = AccumulatedErrorFeedback(quantization_bits)
self .seed_replay = StatelessSeedReplay() if use_seed_replay else None
def training_step (self, batch: dict ) -> dict :
"""Single QES training step."""
estimated_grads = self .grad_estimator.estimate_gradients(
self .model, self .reward_fn, batch
)
total_updates = 0
for name, param in self .model.named_parameters():
if name in estimated_grads:
grad = estimated_grads[name]
_, num_updates = self .error_feedback.apply_accumulated_update(
param.data, name, grad, self .learning_rate
)
total_updates += num_updates
reward = self .reward_fn(self .model, batch)
return {
'reward' : reward,
'num_updates' : total_updates,
'update_ratio' : total_updates / sum (p.numel() for p in self .model.parameters())
}
Step 5: Fine-tuning loop def fine_tune_quantized_llm (
model: torch.nn.Module,
train_loader,
verifier,
num_steps: int = 1000 ,
quantization_bits: int = 4 ,
learning_rate: float = 1e-3 ,
device: str = 'cuda'
):
"""
Fine-tune quantized LLM using QES.
Args:
model: Quantized language model (INT4 or INT8)
train_loader: Training data
verifier: Reward function (task accuracy or loss)
num_steps: Optimization steps
quantization_bits: Bit-width of quantization
learning_rate: ES learning rate
"""
def reward_fn (model, batch ):
"""Compute reward on batch."""
with torch.no_grad():
outputs = model(**batch)
loss = outputs.loss if hasattr (outputs, 'loss' ) else outputs[0 ]
return -loss.item()
qes = QuantizedEvolutionStrategies(
model, reward_fn,
quantization_bits=quantization_bits,
pop_size=32 ,
learning_rate=learning_rate
)
for step in range (num_steps):
batch = next (iter (train_loader))
batch = {k: v.to(device) for k, v in batch.items()}
metrics = qes.training_step(batch)
if (step + 1 ) % 100 == 0 :
print (f"Step {step + 1 } : "
f"Reward={metrics['reward' ]:.4 f} , "
f"Updates={metrics['num_updates' ]} , "
f"UpdateRatio={metrics['update_ratio' ]:.4 f} " )
return model
Practical Guidance When to use : Fine-tuning quantized models (INT4, INT8) where gradient-based methods fail due to discrete rounding
pop_size : 16-64 (larger = better gradient estimates but slower)sigma : 0.005-0.02 (perturbation scale)learning_rate : 1e-4 to 1e-2 (depends on pop_size)quantization_bits : 4 (INT4) or 8 (INT8)
Overcomes gradient stagnation in quantized models
No full-precision shadow parameters (memory efficient)
Population-based approach robust to noise
Error accumulation ensures progress despite discrete constraints
pop_size too small → noisy gradients
learning_rate too high → thrashing in discrete space
Not validating that updates actually improve loss
Using on non-quantized models (wastes computation)
Scaling : Linear in pop_size and model size. Efficient for INT4 models.
Reference Paper: https://arxiv.org/abs/2602.03120
Related work: Evolution strategies, quantization, fine-tuning, Delta-Sigma modulation
Benchmarks: Arithmetic reasoning on INT4 quantized models
