discreteness-diffusion-llm

name	discreteness-diffusion-llm
title	On the Role of Discreteness in Diffusion LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.22630
keywords	["diffusion models","language generation","discrete text","parallel decoding","token dependencies","LLM architecture"]
description	Understand fundamental limitations of applying diffusion to discrete text: position-agnostic corruption ignores linguistic structure, and token-wise training misses multi-token dependencies. Design text diffusion systems satisfying five essential properties: position-aware corruption, dependency-aware training, parallel consistency, linguistic structure respecting, and robust handling of token boundaries.

When to Use This Skill

• Building text-based diffusion models with iterative refinement
• Parallel decoding approaches requiring simultaneous multi-token generation
• Language generation tasks where intermediate denoising should respect text structure
• Research into why continuous diffusion works for images but needs modification for text

When NOT to Use This Skill

• Autoregressive sequence-to-sequence tasks (use traditional LLMs instead)
• Tasks requiring strict left-to-right dependency (diffusion is inherently non-causal)
• Real-time language generation (diffusion is iterative, slow)
• Applications where token-level guarantees are critical (legal, medical)

The Fundamental Problem

Diffusion models excel at images because:

• Spatial structure is preserved during noise corruption
• Neighboring pixels have strong local dependencies
• Denoising naturally respects geometric constraints

But text is fundamentally different:

• Discrete tokens: Each position is a categorical choice, not continuous value
• Long-range dependencies: Word relationships span arbitrary distances
• Position information: Tokens have identity (not just position), creating special structure

Standard diffusion directly applied to text creates two critical failures:

Failure 1: Position-Agnostic Corruption

Standard diffusion applies uniform corruption—each position receives the same noise level regardless of linguistic importance.

Original:  "The cat sat on the mat"
Noise level: same for all positions

Problem: Function words (the, on) are noisy same as content (cat, sat)
Better:   Function words are more robust to noise → less important to denoise

Failure 2: Token Dependency Collapse

During parallel decoding, all tokens are generated simultaneously. But tokens aren't independent:

Tokens at position i and i+1 are generated in parallel:
- Position i generates word "read" (ambiguous: past tense? infinitive?)
- Position i+1 needs this information to correctly generate following word
- But they're denoised in parallel, losing this dependency

Five Essential Properties for Text Diffusion

The paper identifies requirements for effective text diffusion systems:

1. Position-Aware Corruption: Noise should respect linguistic position importance

   • Corrupt function words more heavily (they're redundant)