Self Normalizing Networks

Self-Normalizing Networks

This repository contains tutorial implementations for Self-Normalizing Networks (SNNs) based on Klambauer et al. Preserve the teaching purpose of the examples: show how SELU networks are constructed and compared across TensorFlow and PyTorch implementations.

Minimal Tabular SNN Example

Below is a minimal prototype (see TF_2_x/TABULAR-MLP-SELU.py) for showing SNNs on tabular data: load a small scikit-learn dataset, standardize the features, and train a compact Keras MLP with SELU, LeCun initialization, and AlphaDropout.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from tensorflow import keras
from tensorflow.keras import layers

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0, stratify=y
)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

model = keras.Sequential(
    [
        keras.Input(shape=(X_train.shape[1],)),
        layers.Dense(64, activation="selu", kernel_initializer="lecun_normal"),
        layers.AlphaDropout(0.05),
        layers.Dense(32, activation="selu", kernel_initializer="lecun_normal"),
        layers.Dense(1, activation="sigmoid"),
    ]
)
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.fit(X_train, y_train, epochs=20, batch_size=32, validation_split=0.2)
print(model.evaluate(X_test, y_test, verbose=0))

Core SNN Rules

• Pair SELU activations with LeCun normal initialization.
• Use AlphaDropout when dropout is used in SELU networks.
• In PyTorch, use nn.init.kaiming_normal_(..., mode="fan_in", nonlinearity="linear") for SELU layers; this matches LeCun normal initialization.
• Keep comparisons between ReLU/dropout and SELU/AlphaDropout conceptually clear.
• Do not add BatchNorm to SNN examples unless explicitly requested; it changes the comparison.

Repository Structure

• TF_2_x/ contains current TensorFlow/Keras scripts.
• Pytorch/ contains PyTorch notebooks.
• The root environment.yml is the main environment for current TF2 and PyTorch examples.