Semi Supervised Novelty Detection Eval

semi-supervised-novelty-detection-eval

Semi-supervised novelty detection using ensembles with regularized disagreement — Tifrea et al. (2020) (arXiv:2012.05825, 2020)

What this evaluates

Evaluates a model's ability to distinguish in-distribution (ID) samples from out-of-distribution (OOD) samples in a semi-supervised setting, where only labeled ID data and unlabeled mixed data are available during training.

Datasets

• MNIST — total ?; splits: train (-1), test (-1)
• FashionMNIST — total ?; splits: train (-1), test (-1)
• SVHN — total ?; splits: train (-1), test (-1)
• CIFAR10 — total ?; splits: train (-1), test (-1)
• CIFAR100 — total ?; splits: train (-1), test (-1)
• ImageNet — total ?; splits: train (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• TNR@95 — range: [0, 1]
  • True Negative Rate measured at a fixed False Positive Rate of 5%, indicating the proportion of OOD samples correctly rejected when the ID false alarm rate is 5%.

Input / output format

Input: Image or medical scan samples passed through a neural network to produce class probabilities.

Output: Softmax probability distribution over classes; ensemble score computed as the entropy of the averaged softmax probabilities across models.

Scoring recipe

# predictions: list of softmax outputs per ensemble member for a batch
# gold: binary labels (1 for ID, 0 for OOD)
avg_probs = np.mean(predictions, axis=0)
score = -np.sum(avg_probs * np.log(avg_probs + 1e-10), axis=1)
auroc = roc_auc_score(gold, score)
tnr95 = tnr_at_fpr(gold, score, fpr=0.05)

Common pitfalls

• Pretraining on the full training set in novel-class settings leaks OOD information, artificially inflating detection performance.
• Baselines use heterogeneous architectures; comparing them directly may conflate architectural capacity with algorithmic novelty detection capability.
• PU-learning baselines require the exact OOD ratio in the unlabeled set; using estimated ratios degrades performance and breaks the upper-bound guarantee.

Evidence (verbatim from paper)

For a test sample, we average the outputs of the softmax probabilities predicted by the models, and use the entropy of the resulting distribution as the score for the hypothesis test described in Section 2.3. ... Therefore, we obtain an upper bound on the AUROC/TNR@95.

Citation

@misc{tifrea2020semi,
  title={Semi-supervised novelty detection using ensembles with regularized disagreement},
  author={Tifrea et al. (2020)},
  year={2020},
  note={arXiv:2012.05825}
}

• arXiv: 2012.05825