Improve diffusion model capacity efficiency by directly predicting clean data instead of noise—leverage the manifold assumption that natural data occupies low-dimensional space while noise spans full dimensionality.
Improve diffusion model capacity efficiency by directly predicting clean data instead of noise—leverage the manifold assumption that natural data occupies low-dimensional space while noise spans full dimensionality.
Recover Capacity by Direct Denoising—Predict Clean Data Not Noise
Diffusion models typically predict noise (epsilon) or velocity at each denoising step. This choice seems natural but imposes a hidden cost: noise and velocity distributions span the full data dimensionality, wasting model capacity on high-dimensional structure. Direct x-prediction assumes natural data lies on a low-dimensional manifold—the model only needs to preserve essential information while filtering noise, using capacity more efficiently.
This paper demonstrates that limited-capacity networks can generate high-dimensional data via x-prediction where epsilon/v-prediction fails. The insight is practical: shift the prediction target to align with your data's intrinsic dimensionality, not the ambient space.
Core Concept
Diffusion models work by iteratively adding and removing noise. At each step, the model must predict either:
Noise (ε-prediction): The random noise added; distributed across full dimensionality
Velocity (v-prediction): A mixture; also spreads across full space
Clean data (x-prediction): The denoised output; concentrated on low-dimensional manifold
The mathematical relationships between these are equivalent—they're coordinate transformations of the same underlying process. However, they differ fundamentally in capacity requirement:
High-dimensional noise: Requires network to preserve all directional information; capacity-intensive
Low-dimensional data: Network only needs to capture essential structure; more efficient
This manifold assumption—natural data lives in lower-dimensional space than noise—creates practical capacity gaps observable at scale.
Architecture Overview
Manifold Assumption: Natural data concentrates on low-dimensional manifold; noise fills full high-dimensional space
x-Prediction Head: Network outputs clean data directly, not noise or velocity
Matching Framework: Use standard diffusion loss (MSE) adapted for x-prediction via mathematical conversion
Patch-Based Encoding: For image/video models, use large patches to reduce dimensionality and align with manifold structure
Capacity Scaling: Lower capacity networks exhibit larger performance gaps; x-prediction shows more benefit at limited budgets
Implementation Steps
Step 1: Reformulate Diffusion Process for x-Prediction. Convert noise schedule to x-prediction formulation.
Step 3: Training Loop. Train with x-prediction loss.
deftrain_x_prediction_diffusion(model, dataloader, num_epochs=100, lr=1e-4):
"""
Train diffusion model with x-prediction.
"""
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
diffusion = XPredictionDiffusion(timesteps=1000)
for epoch inrange(num_epochs):
total_loss = 0for x_0 in dataloader: # x_0: clean images
batch_size = x_0.shape[0]
# Random timesteps
t = torch.randint(0, diffusion.timesteps, (batch_size,))
# Forward process: add noise
x_t, noise = diffusion.add_noise(x_0, t)
# Model prediction (x-prediction)
x_0_pred = model(x_t, t)
# Loss: direct MSE between prediction and clean data
loss = diffusion.loss_x_prediction(x_0, t, x_0_pred)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
total_loss += loss.item()
avg_loss = total_loss / len(dataloader)
print(f"Epoch {epoch}: avg_loss={avg_loss:.4f}")
Step 4: Inference (Denoising). Generate by iteratively denoising.
@torch.no_grad()defgenerate_x_prediction(model, diffusion, shape, num_steps=100):
"""
Generate samples via iterative x-prediction denoising.
"""# Start with noise
x_t = torch.randn(shape)
# Iterate denoising (reverse process)
timesteps = torch.linspace(999, 0, num_steps).long()
for t in timesteps:
t_batch = torch.full((x_t.shape[0],), t)
# Model predicts clean image at this step
x_0_pred = model(x_t, t_batch)
# Get alphas for current and next step
alpha_t = diffusion.alphas_cumprod[t]
alpha_prev = diffusion.alphas_cumprod[t - 1] if t > 0else1.0# Update x_t using predicted x_0# This is the reverse step in x-prediction formulation
c1 = torch.sqrt((1 - alpha_prev) / (1 - alpha_t))
c2 = torch.sqrt(alpha_prev / alpha_t)
x_t = c2 * x_0_pred + c1 * (x_t - torch.sqrt(1 - alpha_t) * x_0_pred)
# Add minimal noise if not final stepif t > 0:
noise = torch.randn_like(x_t)
sigma_t = torch.sqrt((1 - alpha_prev) / (1 - alpha_t) * (1 - alpha_t / alpha_prev))
x_t = x_t + sigma_t * noise
return x_t.clamp(-1, 1)
Practical Guidance
When to Use: Training diffusion models on bounded data (images, quantized latents) with limited model capacity. x-Prediction shows largest benefits when model capacity is restricted (< 1B parameters).
Hyperparameters:
Patch size: larger patches (8–16) reduce dimensionality and favor x-prediction; adjust based on image resolution
Schedule: use same beta schedule as noise-pred; mathematical equivalence enables direct comparison
Learning rate: typically same as noise-pred; no special adjustments needed
Pitfalls:
Unbounded predictions: Models can output values outside data range; apply clipping or use bounded activations
Instability at early timesteps: Very noisy inputs may lead to erratic x-0 predictions; use warm-up or gradient clipping
Not always faster: Wall-clock time may be similar (both models have same architecture); gain is capacity efficiency, not compute
Dataset-dependent: Manifold assumption stronger on natural images; weaker on synthetic/high-dimensional data
When NOT to Use: Very large models where capacity is not a bottleneck; data not concentrated on low-dimensional manifold.
Integration: Drop-in replacement for noise-prediction models; use same sampling procedures with adapted formulations.
