| name | sparselora-contextual-sparsity-finetuning |
| title | SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.16500 |
| keywords | ["LoRA","FinetuningAcceleration","Sparsity","Efficiency","LargeLanguageModels"] |
| description | Accelerates LoRA fine-tuning 2.2× computationally and 1.6× wall-clock by leveraging contextual sparsity to compute gradients only for important weight channels. Uses training-free SVD sparsity estimation without full computation. Apply for efficient fine-tuning on memory-constrained GPUs or large-scale training scenarios. |
SparseLoRA: Efficient Fine-Tuning Through Context-Aware Sparse Weight Computation
Large language model fine-tuning with LoRA is efficient compared to full parameters, but still requires computing gradients across all weight channels for every input. SparseLoRA recognizes that different inputs activate different neurons—some channels are critical for certain tokens while irrelevant for others. By dynamically sparsifying gradient computation based on input context, SparseLoRA achieves 2.2× FLOPs reduction and 1.6× wall-clock speedup while preserving model accuracy. The key is a training-free SVD-based sparsity estimator that identifies important channels without expensive forward passes.
The insight is that contextual sparsity (varying importance across input sequences) is more effective than static sparsity. By computing only essential gradients per input, the model maintains expressivity while reducing computation.
Core Concept
SparseLoRA applies contextual sparsity at three dimensions:
- Layer-wise Sparsity: Deeper layers tolerate more sparsity than earlier layers (non-uniform per layer)
- Token-wise Sparsity: Output tokens (targets for loss) require dense computation; context tokens can be sparse
- Step-wise Sparsity: Begin training dense, transition to sparse in later epochs
The framework uses an SVD-based sparsity estimator that:
- Projects inputs through low-rank decompositions of pretrained weights
- Identifies important channels without full gradient computation
- Adds only 0.8% overhead while capturing oracle sparsity patterns
Architecture Overview
- SVD Sparsity Estimator: Decomposes pretrained weights to predict important channels
- Layer-wise Non-uniform Sparsity: Different sparsity levels per layer depth
- Token-wise Selection: Preserves computation for loss targets
- L2 Norm Criterion: For FFN/attention value-output (highest activation magnitudes)
- QK Norm Criterion: For query-key projections (product of normalized scores)
- LoRA Integration: Seamless integration with existing LoRA fine-tuning
Implementation
SVD-based training-free sparsity estimation:
import torch
import torch.nn as nn
import numpy as np
from typing ,
:
():
.model = model
.rank = rank
.svd_decompositions = {}
._precompute_svd()
():
name, module .model.named_modules():
(module, nn.Linear):
W = module.weight.detach().cpu()
U, S, Vt = torch.svd(W)
.svd_decompositions[name] = {
: U[:, :.rank],
: S[:.rank],
: Vt[:.rank, :],
: W.shape
}
() -> [torch.Tensor, ]:
batch_size, seq_len = input_ids.shape
hidden_dim = (.svd_decompositions.values())[][][]
embeddings = .model.get_input_embeddings()(input_ids)
activations = torch.zeros(hidden_dim)
name, svd_data .svd_decompositions.items():
name:
projected = embeddings @ svd_data[].T.cuda()
activation = torch.norm(projected, dim=-).mean(dim=)
activations += activation.cpu()
base_sparsity = ._get_base_sparsity(layer_idx)
token_type == :
sparsity_ratio = base_sparsity *
:
sparsity_ratio = base_sparsity
num_keep = (hidden_dim * ( - sparsity_ratio))
threshold = torch.topk(activations, num_keep)[][-]
sparsity_mask = (activations >= threshold).()
sparsity_mask, sparsity_ratio
() -> :
num_layers =
layer_fraction = layer_idx / num_layers
layer_fraction < :
layer_fraction < :
:
(nn.Module):
():
().__init__()
.model = model
.rank = rank
.lora_alpha = lora_alpha
.sparsity_estimator = sparsity_estimator
.current_step =
.total_steps =
.lora_A = {}
.lora_B = {}
._init_lora()
():
name, module .model.named_modules():
(module, nn.Linear) name name:
in_features = module.in_features
out_features = module.out_features
.lora_A[name] = nn.Parameter(
torch.randn(.rank, in_features) / np.sqrt(.rank)
)
.lora_B[name] = nn.Parameter(torch.zeros(out_features, .rank))
():
outputs = .model(input_ids, labels=labels)
.training labels :
layer_idx =
name, module .model.named_modules():
(module, nn.Linear):
mask, _ = .sparsity_estimator.estimate_sparsity(
input_ids, layer_idx, token_type=
)
(module, ) module.weight.grad :
module.weight.grad = module.weight.grad * mask.view(-, )
layer_idx +=
outputs
():
outputs = .forward(input_ids, labels=labels)
loss = outputs.loss
sparsity_schedule = ._get_sparsity_schedule(training_step)
sparsity_schedule > :
loss.backward(retain_graph=)
name, module .model.named_modules():
(module, nn.Linear) module.weight.grad :
grad_magnitude = torch.(module.weight.grad)
threshold = torch.quantile(
grad_magnitude.flatten(),
sparsity_schedule
)
module.weight.grad = module.weight.grad * (
grad_magnitude > threshold
).()
loss
() -> :
warmup_steps = ( * .total_steps)
step < warmup_steps:
progress = (step - warmup_steps) / (.total_steps - warmup_steps)
max_sparsity =
progress * max_sparsity
():
sparsity_estimator = SVDSparsityEstimator(model, rank=)
sparse_lora = SparseLoRA(
model,
rank=,
sparsity_estimator=sparsity_estimator
)
optimizer = torch.optim.AdamW(sparse_lora.parameters(), lr=learning_rate)
total_steps = (train_loader) * num_epochs
sparse_lora.total_steps = total_steps
model.train()
step =
epoch (num_epochs):
batch train_loader:
input_ids = batch[].cuda()
labels = batch[].cuda()
loss = sparse_lora.compute_sparse_loss(input_ids, labels, step)
optimizer.zero_grad()
loss.backward()
optimizer.step()
step +=
step % == :
()
model
