ddit-dynamic-diffusion-patch-scheduling

name	ddit-dynamic-diffusion-patch-scheduling
title	DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.16968
keywords	["diffusion models","efficiency optimization","dynamic inference","patch scheduling","image generation"]
description	Accelerate diffusion transformer inference by dynamically adjusting patch granularity during generation based on detail complexity at each timestep. Early denoising steps (establishing low-frequency structure) use coarse patches; later steps (adding high-frequency detail) use fine patches. Achieves 3.52× speedup on FLUX-1.Dev and 3.2× on video models while maintaining quality through variance-based adaptive scheduling.

DDiT: Variance-Aware Adaptive Patch Sizing for Efficient Diffusion

Diffusion transformers achieve state-of-the-art generation quality but process every diffusion timestep with uniform computational cost. However, timesteps serve fundamentally different purposes: early steps establish global structure (low-frequency components), while late steps refine texture and detail (high-frequency). Using fixed patch granularity throughout is computationally wasteful—coarse patches suffice early, but fine patches become necessary only late in generation.

Traditional approaches use fixed patch sizes throughout inference. The challenge is determining when detail-level refinement becomes necessary without human tuning or expensive online measurements.

Core Concept

DDiT uses the rate of change in the latent manifold as a proxy for detail complexity, enabling automatic scheduling decisions. The approach measures acceleration (third-order finite differences) of the latent representation over timesteps. High acceleration indicates rapid latent evolution and suggests detail generation is active; low acceleration indicates stable structure and allows coarser patches.

The system divides latents into spatial regions and compares acceleration variance against a threshold. Regions with low variance use large patch sizes; high-variance regions use fine patches.

Architecture Overview

• Variable Patch Support: Modify patch embedding to support multiple granularities (p, 2p, 4p) using LoRA-style adaptation branches
• Latent Tracker: Monitor latent representations across recent timesteps to compute second and third-order finite differences
• Variance Measurer: Compute acceleration variance within spatial patches
• Scheduler: Compare variance to threshold; select patch size per region
• Efficient Router: Use coarse patches early, progressively refine as variance increases

Implementation

Implement multi-scale patch support by adding LoRA branches to the patch embedding layer:

class AdaptivePatchEmbedding(nn.Module):
    def __init__(self, dim, patch_size=16):
        super().__init__()
        self.patch_size = patch_size
        self.base_embedding = nn.Linear(3 * patch_size**2, dim)

        
        .coarse_2x = nn.Sequential(
            nn.Linear( * ( * patch_size)**, dim // ),
            nn.Linear(dim // , dim)
        )
        .coarse_4x = nn.Sequential(
            nn.Linear( * ( * patch_size)**, dim // ),
            nn.Linear(dim // , dim)
        )

     ():
        
         patch_scale == :
            
            patches = ._extract_patches(x, .patch_size)
             .base_embedding(patches)
         patch_scale == :
            patches = ._extract_patches(x, .patch_size * )
             .coarse_2x(patches)
         patch_scale == :
            patches = ._extract_patches(x, .patch_size * )
             .coarse_4x(patches)

     ():
        
        B, C, H, W = x.shape
        patches = x.reshape(
            B, C, H // patch_size, patch_size,
            W // patch_size, patch_size
        )
        patches = patches.permute(, , , , , ).reshape(
            B, -, C * patch_size * patch_size
        )
         patches