jit-spatial-diffusion-acceleration

name	jit-spatial-diffusion-acceleration
title	Just-in-Time: Training-Free Spatial Acceleration for Diffusion Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.10744
keywords	["Diffusion","Acceleration","Spatial Tokens","Inference","Transformers"]
description	Accelerate diffusion transformers by processing only sparse anchor tokens in early stages, then expanding to full spatial resolution using learnable extrapolation. Combines SAG-ODE velocity field lifting with importance-guided token activation for lossless speedup.

Technique: Spatial Token Approximation via Importance-Guided Anchor Selection

Diffusion transformers generate images by iteratively denoising all spatial tokens across T timesteps. This full-token processing is computationally expensive. Just-in-Time (JiT) accelerates inference by computing only anchor tokens initially, then extrapolating the velocity field to predict evolution of non-anchor tokens, finally expanding to full resolution in later stages.

The key insight is that early diffusion steps contain redundancy: full spatial processing is unnecessary before the model establishes coarse structure. By selecting anchors based on velocity variance, the method preserves high-activity regions while approximating others.

Core Concept

JiT operates across three complementary mechanisms:

1. Spatially Approximated Generative ODE (SAG-ODE): Uses an augmented lifter operator to extrapolate velocity fields from anchor tokens to full space.

2. Deterministic Micro-Flow (DMF): Ensures smooth transitions when expanding token sets, maintaining consistency between stages.

3. Importance-Guided Token Activation (ITA): Dynamically selects tokens based on local velocity variance rather than fixed patterns.

This enables stage-adaptive token reduction: aggressive sparsity early when structure is coarse, gradual token addition as generation details accumulate.

Architecture Overview

• Anchor token selector: Identifies sparse subset based on velocity variance
• Lifter operator: Maps sparse velocity field to full token space
• DMF handler: Manages token set expansion with smooth transitions
• Full transformer layers: Operate on growing token set over timesteps
• Lossless design: No upsampling artifacts or reconstruction losses

Implementation Steps

Step 1: Compute Anchor Tokens via ITA

Select tokens based on local velocity variance in the generative ODE.

import torch
import torch.nn.functional as F

def select_anchor_tokens(velocity_field, anchor_ratio=0.25):
    """
    Select anchor tokens based on local velocity variance.

    velocity_field: (batch, seq_len, dim) velocity predictions from transformer
    anchor_ratio: fraction of tokens to retain as anchors
    """
    
    
    kernel_size = 
    padding = kernel_size // 

    
    unfolded = F.unfold(
        velocity_field.unsqueeze(-).permute(, , , ),
        kernel_size=(kernel_size, ),
        padding=(padding, )
    )  

    variance = unfolded.std(dim=)  

    
    num_anchors = (anchor_ratio * velocity_field.shape[])
    anchor_indices = torch.topk(variance, num_anchors, dim=)[]

     anchor_indices, variance