nag-diffusion-guidance

name	nag-diffusion-guidance
title	Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.21179
keywords	["diffusion models","negative guidance","attention","few-step sampling","image generation"]
description	Apply training-free negative guidance in diffusion models by extrapolating in attention space with L1-based normalization, restoring suppression of unwanted attributes across architectures and modalities.

Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models

Core Concept

Normalized Attention Guidance (NAG) addresses the breakdown of negative guidance (suppressing unwanted attributes) in diffusion models, particularly in aggressive few-step sampling regimes. While Classifier-Free Guidance (CFG) works well for positive guidance, negative guidance becomes unstable as sampling steps decrease.

NAG introduces a training-free mechanism that performs extrapolation in attention space with L1-based normalization and refinement. By operating at the attention level rather than the latent level, NAG maintains effectiveness across diverse architectures (UNet, DiT), sampling strategies, and modalities (image, video) without requiring model retraining or architecture modifications.

Architecture Overview

• Attention-Space Extrapolation: Extract attention maps from unconditional paths and extrapolate them to suppress unwanted patterns
• L1-Based Normalization: Normalize attention extrapolations using L1 distance metrics for stability
• Refinement Pipeline: Apply post-processing to smooth and validate guidance signals
• Plug-in Integration: Works with any diffusion architecture by intercepting attention computations
• Multi-Modal Support: Applies uniformly to image, video, and other modalities
• Minimal Computational Overhead: Negligible impact on inference speed compared to standard sampling

Implementation

The following steps outline how to integrate NAG into a diffusion sampling pipeline:

1. Extract baseline attention maps - Run unconditional generation and capture intermediate attention states
2. Compute guidance direction - Calculate L1-normalized difference between conditional and unconditional paths
3. Perform attention extrapolation - Extrapolate guidance signals in attention space beyond typical guidance scales
4. Apply refinement - Smooth and validate the extrapolated attention to prevent artifacts
5. Integrate into sampling loop - Inject guidance at appropriate denoising steps
6. Monitor quality - Track text alignment and fidelity metrics during generation

import torch
import torch.nn.functional as F
from typing import , 

 :
     ():
        .guidance_scale = guidance_scale
        .l1_epsilon = l1_epsilon

     () -> [, torch.Tensor]:
        
        attention_maps = {}
         name, module  model_output.items():
               name:
                attention_maps[name] = module.detach()
         attention_maps

     () -> torch.Tensor:
        
        diff = cond_attn - uncond_attn
        l1_norm = torch.norm(diff, p=, dim=-, keepdim=)
        l1_norm = torch.clamp(l1_norm, =.l1_epsilon)
        normalized_guidance = diff / l1_norm
         normalized_guidance

     () -> torch.Tensor:
        
        extrapolated = guidance * .guidance_scale
         extrapolated

     () -> torch.Tensor:
        
        
        kernel = torch.ones(, , smoothing_window, smoothing_window) / (smoothing_window ** )
        kernel = kernel.to(attention.device)
        refined = F.conv2d(attention.unsqueeze(), kernel, padding=smoothing_window//)
        refined = refined.squeeze()

        
        refined = torch.clamp(refined, , )
         refined

     () -> torch.Tensor:
        
        guided_latents = model_latents.clone()

         name  uncond_attn.keys():
            guidance = .compute_l1_normalized_guidance(uncond_attn[name], cond_attn[name])
            guidance = .extrapolate_guidance(guidance)
            guidance = .refine_attention(guidance)

         guided_latents