nbdiff-block-diffusion-llm

name	nbdiff-block-diffusion-llm
title	From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.06776
keywords	["diffusion language models","block generation","auto-regressive to diffusion","parallel decoding","model adaptation"]
description	Convert auto-regressive language models to efficient diffusion-based generators through gradual block size increments. NBDiff-7B inherits long-context capabilities from AR predecessors while achieving state-of-the-art parallel generation—ideal when you need efficiency without sacrificing reasoning.

Overview

The paper proposes a principled pathway for converting AR models to diffusion language models through the Block-Diffusion paradigm. Rather than ad-hoc modifications, the approach gradually transitions from single-token to block-based generation while maintaining computational efficiency and preserving AR model capabilities like long-context and reasoning.

When to Use

• Adapting existing autoregressive models to diffusion-based generation
• Scenarios requiring parallel decoding without sequential bottlenecks
• Long-context models needing efficiency improvements
• Applications where reasoning capabilities must be preserved
• Transitioning production AR models to faster inference

When NOT to Use

• Models already achieving acceptable inference speed
• Sequential dependencies where parallelization fails
• Scenarios where diffusion generation quality lags behind AR
• Applications without engineering resources for model conversion
• Real-time systems where adaptation overhead is unacceptable

Core Technique

Principled AR-to-DLM adaptation through gradual block size progression:

# Principled adaptation from AR to Diffusion Language Models
class ARToDiffusionAdapter:
    def __init__(self, ar_model):
        """
        Initialize adapter with pre-trained autoregressive model.
        Preserve all existing capabilities during conversion.
        """
        self.ar_model = ar_model
        self.block_size = 1  # Start with single tokens

    def gradual_block_size_increment(self, training_stages=3):
        """
        Progression from block size 1 (AR) to full blocks.
        Maintains training stability and efficiency.
        """
        for stage in range(training_stages):
            # Gradually increase block size
            self.block_size =  ** stage  

            
            .train_block_diffusion(.block_size)

            ()

         .ar_model

     ():
        
         batch  .training_loader:
            
            x = batch[]
            seq_len = x.shape[]

            
            blocks = .reshape_to_blocks(x, block_size)

            
            noise = torch.randn_like(blocks)
            t = torch.randint(, , (blocks.shape[],))
            noisy_blocks = .add_noise(blocks, noise, t)

            
            pred_blocks = .ar_model(noisy_blocks, t)

            
            loss = torch.nn.functional.mse_loss(pred_blocks, blocks)
            loss.backward()

            .optimizer.step()

     ():
        
        batch_size, seq_len = x.shape

        
        pad_len = (block_size - (seq_len % block_size)) % block_size
        x_padded = torch.nn.functional.pad(x, (, pad_len))

        
        num_blocks = x_padded.shape[] // block_size
        blocks = x_padded.reshape(batch_size, num_blocks, block_size)

         blocks

     ():
        
        
        
        attention_mask = .create_hybrid_attention_mask()
         attention_mask

     ():
        
        
        context_size = .ar_model.context_window
        context_mask = torch.tril(
            torch.ones(context_size, context_size, dtype=torch.)
        )

        
        
        gen_block_size = .block_size
        gen_mask = torch.eye(gen_block_size, dtype=torch.).unsqueeze()

        
        full_mask = torch.block_diag(context_mask, gen_mask)
         full_mask

     ():
        
        
        ar_logits = .ar_model(input_ids)

        
        
        initial_blocks = torch.argmax(ar_logits, dim=-)

        
        current_blocks = initial_blocks
         diffusion_step  (num_diffusion_steps):
            
            denoised = .ar_model.denoise(
                noisy_blocks=current_blocks,
                guidance=ar_logits,
                step=diffusion_step
            )
            current_blocks = denoised

         current_blocks

     ():
        
        input_ids = .tokenize(prompt)

        
        num_blocks = max_tokens // .block_size

        
        generated_blocks = []

         block_idx  (num_blocks):
            
            
            context = torch.cat(
                [input_ids] + generated_blocks,
                dim=
            )

            
            block = .generate_block(context)
            generated_blocks.append(block)

        
        output_ids = torch.cat(
            [input_ids] + generated_blocks,
            dim=
        )

         .detokenize(output_ids)

     ():
        
        
        
        

        
        performance = .evaluate_on_long_context_tasks()
         performance

     ():
        
        
        
        
        

        
        performance = .evaluate_on_reasoning_tasks()
         performance