derf-normalization-free-transformers

name	derf-normalization-free-transformers
title	Stronger Normalization-Free Transformers
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.10938
keywords	["normalization-free","transformers","activation functions","generalization","training stability"]
description	Replace LayerNorm with Derf(x) = erf(αx + s) for improved generalization in transformers. Derf outperforms LayerNorm across vision, speech, and DNA modeling—ideal when normalization-free training provides benefits without architectural complexity.

Overview

The paper demonstrates that simpler point-wise functions can replace traditional normalization while achieving superior generalization. Derf, based on rescaled Gaussian CDF, provides normalization-free transformers with improved performance across diverse domains.

When to Use

• Training transformers without normalization layers
• Scenarios where generalization is critical
• Vision, speech, and sequence modeling tasks
• Need for improved training stability
• Replacing LayerNorm or RMSNorm

When NOT to Use

• Applications already achieving good normalization
• Scenarios where norm-dependent features matter
• Models requiring specific normalization properties

Core Technique

Derf activation function design:

# Normalization-free transformers with Derf
class DerfNormalizationFree(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.alpha = nn.Parameter(torch.ones(1))
        self.s = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        """Derf(x) = erf(α*x + s)"""
        from scipy.special import erf as scipy_erf

        # Apply Derf activation
        output = torch.erf((self.alpha * x + self.s) / math.sqrt(2.0))

        return output

class TransformerBlock(nn.Module):
    def __init__(self, dim, num_heads):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, num_heads)
        
        .derf = DerfNormalizationFree(dim)
        .ffn = nn.Sequential(
            nn.Linear(dim,  * dim),
            .derf,
            nn.Linear( * dim, dim)
        )

     ():
        
        attn_output = .attn(x, x, x)[]
        
        x = x + .derf(attn_output)

        
        ffn_output = .ffn(x)
        
        x = x + .derf(ffn_output)

         x