# Derf Normalization Free Transformers

> Replace LayerNorm with Derf(x) = erf(αx + s) for improved generalization in transformers. Derf outperforms LayerNorm across vision, speech, and DNA modeling—ideal when normalization-free training provides benefits without architectural complexity.

- Skill: `adu2021/derf-normalization-free-transformers` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/derf-normalization-free-transformers`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/derf-normalization-free-transformers/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/derf-normalization-free-transformers

---


## Overview

The paper demonstrates that simpler point-wise functions can replace traditional normalization while achieving superior generalization. Derf, based on rescaled Gaussian CDF, provides normalization-free transformers with improved performance across diverse domains.

## When to Use

- Training transformers without normalization layers
- Scenarios where generalization is critical
- Vision, speech, and sequence modeling tasks
- Need for improved training stability
- Replacing LayerNorm or RMSNorm

## When NOT to Use

- Applications already achieving good normalization
- Scenarios where norm-dependent features matter
- Models requiring specific normalization properties

## Core Technique

Derf activation function design:

```python
# Normalization-free transformers with Derf
class DerfNormalizationFree(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.alpha = nn.Parameter(torch.ones(1))
        self.s = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        """Derf(x) = erf(α*x + s)"""
        from scipy.special import erf as scipy_erf

        # Apply Derf activation
        output = torch.erf((self.alpha * x + self.s) / math.sqrt(2.0))

        return output

class TransformerBlock(nn.Module):
    def __init__(self, dim, num_heads):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, num_heads)
        # NO LayerNorm, use Derf instead
        self.derf = DerfNormalizationFree(dim)
        self.ffn = nn.Sequential(
            nn.Linear(dim, 4 * dim),
            self.derf,
            nn.Linear(4 * dim, dim)
        )

    def forward(self, x):
        # Self-attention
        attn_output = self.attn(x, x, x)[0]
        # Apply Derf instead of LayerNorm
        x = x + self.derf(attn_output)

        # Feed-forward
        ffn_output = self.ffn(x)
        # Derf-based combination
        x = x + self.derf(ffn_output)

        return x
```

## Key Results

- Derf outperforms LayerNorm on generalization
- Effective across vision, speech, DNA
- Dynamic Tanh surpassed
- Improved training stability

## References

- Original paper: https://arxiv.org/abs/2512.10938
- Focus: Normalization-free transformer training
- Domain: Transformer architectures, activation functions