adaptive-lora-personalized-ranks

name	adaptive-lora-personalized-ranks
title	Not All Layers Are Created Equal: Adaptive LoRA Rank Allocation for Personalized Image Generation
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.21884
keywords	["Low-Rank Adaptation","Adaptive Ranks","Diffusion Models","Parameter Efficiency","Personalization"]
description	Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.

Adaptive LoRA: Per-Layer Rank Allocation

Problem Statement

Current LoRA practice applies identical ranks uniformly across all layers, which is suboptimal because layer importance varies and subjects have different complexity. Simple subjects waste capacity with high-rank components, while complex subjects suffer from insufficient expressiveness with uniform low ranks.

Component Innovation: Adaptive Rank Learning

The Modification: Replace fixed-rank LoRA with learnable per-layer rank parameters (νℓ) that are optimized during fine-tuning.

Technical Mechanism:

The method introduces learnable parameters νℓ for each LoRA component that control effective rank. A discretized exponential distribution imposes importance ordering on rank indices, preventing all ranks from collapsing identically.

# Variational rank framework: learnable importance weights per layer
# For each layer l, learn importance parameters nu_l that gate
# which rank dimensions activate during training
# Rank is dynamically added/removed via gated forward passes

# Forward pass with adaptive rank masking:
# out = (A @ diag(Lambda_l) @ B @ x)
# where Lambda_l is dynamically scaled based on learned nu_l

Weight rescaling through diagonal matrices (Λℓ) normalizes magnitudes during forward passes. The training loss combines three objectives:

• Reconstruction loss (MSE between fine-tuned and target outputs)
• Rank regularization (pushing toward target rank)
• Cross-attention entropy minimization (preserving attention patterns)

Ablation Results

Memory Footprint: LoRA2 achieves comparable visual quality with 0.40 GB vs. 2.80 GB for fixed rank-512 LoRA—a 7× reduction.

Quality Metrics across 29 subjects:

• Subject fidelity (DINO, CLIP-I): Competitive with fixed-rank baselines
• Text alignment (CLIP-T): Better than many fixed-rank configurations
• Rank analysis: Optimal ranks vary significantly across subjects (range 32-256) and layers, confirming heterogeneous requirements

Drop-In Checklist

1. Initialization: Start with fixed rank estimate (e.g., 64) across all layers
2. Enable Adaptation: Introduce learnable νℓ parameters with exponential prior
3. Training: Use three-component loss; tune regularization strength via validation