t-lora-single-image-diffusion-customization

name	t-lora-single-image-diffusion-customization
title	T-LoRA: Single Image Diffusion Model Customization Without Overfitting
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.05964
keywords	["Diffusion Models","Single-Image Personalization","LoRA","Timestep-Dependent Adaptation","Orthogonal Initialization"]
description	Personalize diffusion models to learn a concept from a single image without overfitting by using timestep-dependent rank masking and orthogonal weight initialization, enabling faithful concept reproduction while maintaining text-guided control.

T-LoRA: Timestep-Dependent Personalization of Diffusion Models

Training diffusion models on a single concept image is notoriously difficult. Standard LoRA quickly overfits: the model memorizes the exact image instead of learning the concept. The root cause is that different denoising timesteps have different susceptibility to overfitting. Early timesteps (high noise) have natural regularization. Late timesteps (low noise, near-pixel level) are prone to memorizing exact pixel patterns. T-LoRA addresses this by dynamically adjusting adapter rank as a function of timestep: fewer parameters at noisy timesteps, more at clean ones.

A complementary insight: standard LoRA matrices accumulate linear dependence that undermines selective activation. T-LoRA applies orthogonal initialization using SVD components, ensuring that masked rank subsets remain effective for meaningful feature learning.

Core Concept

The challenge is learning a visual concept (a person, object, style) from one image while keeping the model generalizable. T-LoRA exploits the fact that the denoising process has natural structure: early timesteps handle coarse structure and are hard to overfit, late timesteps handle details and are easy to overfit. By allocating fewer adapter parameters early and more late, the model can focus on learning genuine concept features rather than pixel memorization.

Orthogonal weight initialization complements this by ensuring that even when you mask out certain rank components, the remaining ones stay active and meaningful. Without orthogonality, masking can create dead components.

Architecture Overview

• Timestep-Dependent Rank Function: r(t) = floor((r - r_min) * (T - t) / T) + r_min, allocating higher rank at lower timesteps
• Rank Masking: Diagonal matrices selectively activate adapter components during training and inference
• Orthogonal LoRA Initialization: Decompose random matrices via SVD, use lower singular components to avoid overfitting
• Integrated T-LoRA: Combines both mechanisms for optimal performance
• Target Models: Tested on Stable Diffusion-XL and FLUX-1.dev

Implementation

Step 1: Analyze Timestep-Specific Overfitting

First, diagnose which timesteps overfit most by training with fixed-rank adapters and measuring concept fidelity vs. text alignment:

import torch
import torch.nn as nn
from diffusers import StableDiffusionXLPipeline
from peft import get_peft_model, LoraConfig

 ():
    
    pipeline = StableDiffusionXLPipeline.from_pretrained(
        
    )
    unet = pipeline.unet

    
    lora_config = LoraConfig(
        r=,
        lora_alpha=,
        target_modules=[, ],
        lora_dropout=,  
        bias=
    )
    lora_unet = get_peft_model(unet, lora_config)

    
    optimizer = torch.optim.Adam(lora_unet.parameters(), lr=)

     step  ():
        
        timesteps = torch.randint(, num_timesteps, (,))
        noisy_image = add_noise_to_concept(concept_image, timesteps[])

        
        noise_pred = lora_unet(
            noisy_image,
            timesteps[],
            encoder_hidden_states=encode_text(concept_name, pipeline)
        ).sample

        
        loss = F.mse_loss(noise_pred, torch.randn_like(noise_pred))
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

    
    fidelity_scores = {}
    alignment_scores = {}

     t  (, num_timesteps, ):
        
        samples_t = generate_from_concept(
            lora_unet, pipeline,
            concept_name,
            timestep_focus=t,
            num_samples=
        )

        
        concept_score = measure_clip_similarity(
            samples_t, concept_image
        )
        fidelity_scores[t] = concept_score

        
        random_prompts = [, , ]
        alignment = measure_prompt_alignment(samples_t, random_prompts)
        alignment_scores[t] = alignment

    
    (, fidelity_scores)
    (, alignment_scores)
    ()

     fidelity_scores, alignment_scores

 ():
    
     diffusers.schedulers  DDPMScheduler
    scheduler = DDPMScheduler.from_config(scheduler_config  {})

    noise = torch.randn_like(image)
    noisy = scheduler.add_noise(image, noise, timestep)
     noisy