Squashing Activation Eval

squashing-activation-eval

Squashing activation functions in benchmark tests: towards eXplainable Artificial Intelligence using continuous-valued logic — Zeltner et al. (2020) (arXiv:2010.08760, 2020)

What this evaluates

Evaluates a novel 'Squashing' activation function against standard alternatives (ReLU, Sigmoid, Tanh) on synthetic 2D classification tasks and the Fashion-MNIST image classification benchmark. It measures how well continuously differentiable logical approximations perform compared to conventional non-linearities in terms of convergence speed and final classification accuracy.

Datasets

• Fashion-MNIST — total 70000; splits: train (60000), test (10000)
• Synthetic 2D Classification — total 500; splits: train (-1), test (-1)

Metrics

• test accuracy (primary) — range: percent
  • Percentage of correctly classified instances in the held-out test set.
• cross-entropy loss — range: [0, ∞)
  • Negative log-likelihood averaged over the batch, used as the optimization objective.
• train accuracy — range: percent
  • Percentage of correctly classified instances in the training set.

Input / output format

Input: 2D spatial coordinates (x, y) for synthetic tasks; 28×28 grayscale pixel arrays for Fashion-MNIST.

Output: Binary class label (0 or 1) for synthetic tasks; 10-class categorical label (0–9) for Fashion-MNIST.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_cross_entropy(predictions, gold_labels):
    # predictions: softmax probabilities, gold_labels: one-hot encoded
    return -sum(g[i] * log(predictions[i]) for i in range(len(gold_labels))) / len(gold_labels)

Common pitfalls

• The Squashing function with a learnable β parameter requires more initial epochs to converge, which can skew early training comparisons if fixed epoch counts are used.
• The synthetic datasets lack a specified train/test split ratio in the text, making exact replication of the reported train/test accuracy difficult without the original data generation script.
• Static β and dynamic β configurations for the Squashing function are evaluated as separate runs but share identical baseline hyperparameters, potentially obscuring the true computational overhead of the learnable parameter.

Evidence (verbatim from paper)

The dataset is composed of two balanced classes, each containing 250 points. ... FASHION-MNIST, a dataset consisting of 60000 training images and 10000 test images. ... As a cost function cross-entropy function is applied ... train accuracy | test accuracy

Citation

@misc{zeltner2020squashing,
  title={Squashing activation functions in benchmark tests: towards eXplainable Artificial Intelligence using continuous-valued logic},
  author={Zeltner et al. (2020)},
  year={2020},
  note={arXiv:2010.08760}
}

• arXiv: 2010.08760