Nemotron 3 Nano

Overview

Nemotron 3 Nano achieves efficiency through sparse mixture-of-experts combined with hybrid Mamba-Transformer architecture, backed by sophisticated post-training methodology.

Core Technique

Sparse MoE with Hybrid Architecture: 31.6B total parameters, 3.2B active per token (20% activation).

class NemotronNanoMoE:
    def __init__(self):
        # 6-of-128 expert selection
        self.num_experts = 128
        self.top_k = 6  # Activate only 6 experts per token
        self.router = nn.Linear(hidden_dim, 128)
        self.experts = nn.ModuleList([MambaTransformerBlock() for _ in range(128)])

    def forward(self, x):
        # Router selects top-6 experts
        router_logits = self.router(x)
        expert_indices = torch.topk(router_logits, k=6, dim=-1).indices

        # Activate selected experts only
        output = 0
        for idx in range(6):
            expert = self.experts[expert_indices[:, idx]]
            output = output + expert(x)

        return output / 6

Three-Stage Post-Training:

def nemotron_nano_training(base_model):
    # Stage 1: Supervised Fine-Tuning
    model = sft_training(base_model, reasoning_data)

    # Stage 2: Verifiable Reinforcement Learning
    model = rlvr_training(model)

    # Stage 3: RLHF with Length Control
    model = rlhf_with_length_control(model)

    return model

Group Relative Length Control: Prevent excessive reasoning token generation while maintaining quality.

def group_relative_length_reward(trajectory, length_penalty=0.1):
    """
    Length-normalized reward adjustment prevents models from
    inflating reasoning token counts.
    """
    accuracy_reward = 1.0 if trajectory.correct else 0.0
    length_normalized = accuracy_reward / (1.0 + length_penalty * trajectory.num_tokens)
    return length_normalized

When to Use This Technique

Use when: Deploying efficient models at scale, inference throughput critical, agentic reasoning needed, long-context (1M) support required.

When NOT to Use This Technique

Avoid if: Max accuracy required, specialized domain needs full parameter activation.

Implementation Notes

Requires: 128 expert modules, router network, RLVR training, length-controlled reward, 1M token support.

Key Performance

• 3.3× higher throughput than competitors
• 31.6B parameters, 3.2B active
• Competitive reasoning despite sparsity
• 1M token context support

References

• Sparse MoE with 6-of-128 expert selection
• Hybrid Mamba-Transformer blocks as experts
• Three-stage post-training methodology
• Group Relative Length Control for RL