stepsize-learning-rate

name	stepsize-learning-rate
title	Stepsize anything: A unified learning rate schedule for budgeted-iteration training
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.24452
keywords	["learning rate schedule","optimization","convergence","budget-aware","unified framework"]
description	Improve training efficiency under iteration budgets using the Unified Budget-Aware (UBA) schedule, a theoretically grounded learning rate approach governed by a single hyperparameter φ that balances adaptability across network architectures.

Stepsize Anything: Unified Learning Rate Schedule for Budgeted-Iteration Training

Core Concept

Stepsize Anything introduces the Unified Budget-Aware (UBA) learning rate schedule, addressing a practical gap in modern deep learning: how to set learning rates when training is constrained by a fixed iteration budget rather than convergence criteria.

Traditional learning rate schedules assume unlimited computational resources and optimize for eventual convergence. In contrast, UBA is theoretically grounded in landscape curvature properties and explicitly accounts for predetermined computational constraints. The key innovation is expressing the schedule through a single interpretable hyperparameter φ that relates to the problem's condition number, enabling consistent performance across diverse architectures without extensive hyperparameter tuning.

Architecture Overview

• Condition Number Connection: Theoretical link between hyperparameter φ and landscape curvature (condition number)
• Budget-Aware Framework: Explicitly incorporates iteration budget into schedule design rather than treating it as secondary
• Convergence Guarantees: Proven convergence properties for different φ values
• Unified Applicability: Single schedule works across CNNs, Transformers, and other architectures
• Adaptivity: Automatically adjusts strategy based on available budget and problem properties
• Minimal Tuning: Single hyperparameter φ replaces extensive per-task hyperparameter search

Implementation

The following steps outline how to implement and use the UBA schedule:

1. Estimate condition number - Assess landscape curvature characteristics of your problem
2. Set hyperparameter φ - Choose φ value based on condition number estimate and risk tolerance
3. Initialize learning rate schedule - Compute initial lr based on budget and φ
4. Apply schedule during training - Update learning rate according to UBA formula at each step
5. Monitor convergence - Track loss and validation metrics to validate choice of φ
6. Adapt if needed - Adjust φ dynamically if budget changes or convergence is suboptimal

import torch
import torch.optim as optim
from typing import Callable
 math

 :
     ():
        
        .optimizer = optimizer
        .total_steps = total_steps
        .phi = phi
        .base_lr = base_lr
        .condition_number = condition_number
        .current_step = 

     () -> :
        
         .total_steps <= :
             .base_lr

        
        progress = .current_step / .total_steps

        
        
        schedule_value = ( - progress * .phi) ** 

        
        adaptive_factor =  / math.sqrt(.condition_number)

        lr = .base_lr * schedule_value * adaptive_factor
         lr

     ():
        
        lr = .get_learning_rate()
         param_group  .optimizer.param_groups:
            param_group[] = lr
        .current_step += 

     () -> :
        
         steps  :
            steps = .total_steps
        values = []
         step  (steps):
            .current_step = step
            values.append(.get_learning_rate())
         values


 :
     ():
        .model = model
        .train_loader = train_loader
        .device = device

     ():
        
        optimizer = optim.SGD(.model.parameters(), lr=base_lr)
        scheduler = UBAScheduler(optimizer, total_budget, phi=phi,
                                base_lr=base_lr, condition_number=condition_number)
        loss_fn = torch.nn.CrossEntropyLoss()

        total_steps = 
         epoch  (epochs):
             batch_idx, (data, target)  (.train_loader):
                 total_steps >= total_budget:
                    ()
                    

                data, target = data.to(.device), target.to(.device)
                optimizer.zero_grad()

                output = .model(data)
                loss = loss_fn(output, target)
                loss.backward()
                optimizer.step()
                scheduler.step()

                total_steps += 
                 total_steps %  == :
                    lr = scheduler.get_learning_rate()
                    ()

         total_steps