multimodal-diffusion-alignment

name	multimodal-diffusion-alignment
title	Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.07986
keywords	["diffusion-models","text-to-image","multimodal","cross-modal-attention","lora-finetuning"]
description	Improve text-image alignment in diffusion transformers through Temperature-Adjusted Cross-modal Attention (TACA), addressing token imbalance and timestep-dependent weighting with parameter-efficient LoRA fine-tuning.

Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers

Core Concept

Multimodal Diffusion Transformers (MM-DiTs) struggle with text-image alignment due to two fundamental issues: visual tokens vastly outnumber text tokens, causing text guidance to be diluted in attention softmax computation, and attention weights remain static across denoising timesteps despite varying interaction importance. Early denoising prioritizes layout establishment (requiring strong text guidance), while later steps focus on detail refinement. Temperature-Adjusted Cross-modal Attention (TACA) solves both problems through temperature scaling that rebalances modal competition and timestep-dependent weighting that adapts interaction strength across denoising phases.

Architecture Overview

• Modality-Specific Temperature Scaling: Amplifies cross-modal attention logits by factor γ > 1
• Timestep-Dependent Adjustment: Applies temperature only during early denoising (t ≥ t_thresh)
• Piecewise Temperature Function: γ(t) = γ for early steps, 1.0 for detail refinement
• LoRA Fine-tuning: Low-rank adaptation to attention layers for artifact suppression
• Parameter Efficiency: <5% additional parameters, compatible with frozen base models

Implementation

Step 1: Analyze Modal Imbalance in MM-DiT

Understand the core problem before implementing TACA:

import torch
import torch.nn as nn
import torch.nn.functional as F

class ModalImbalanceAnalyzer:
    """Analyze how visual tokens suppress text guidance"""

    def __init__(self, mm_dit_model):
        self.model = mm_dit_model

    def analyze_softmax_suppression(self, text_ids, image_ids, timestep):
        """
        Demonstrate how visual token abundance suppresses text attention.

        In standard softmax, text token contribution is diluted by visual tokens.
        """
        batch_size = text_ids.shape[0]
        num_text_tokens = text_ids.shape[]
        num_image_tokens = image_ids.shape[]  

        
         torch.no_grad():
            embeddings = .model.embed_tokens(text_ids, image_ids, timestep)
            

            hidden = .model.forward_with_hooks(embeddings, return_attention=)
            

        
        attention_logits = hidden[]
        

        
        standard_probs = F.softmax(attention_logits, dim=-)

        
        text_prob_per_position = standard_probs.mean(dim=(, ))  
        text_contribution = text_prob_per_position.().item()

        ()
        ()
        ()

         {
            : attention_logits,
            : text_contribution
        }

     ():
        
        result = .analyze_softmax_suppression(text_ids, image_ids, timestep)

        expected =  / ( + )  
        actual = result[]
        imbalance = (, (expected - actual) / expected)

         imbalance