Goad Anomaly Detection Eval

goad-anomaly-detection-eval

Classification-Based Anomaly Detection for General Data — Bergman et al. (2020) (arXiv:2005.02359, 2020)

What this evaluates

Evaluates a self-supervised, classification-based anomaly detection framework (GOAD) on its ability to distinguish normal data from anomalies without labeled anomalies during training. It probes the model's robustness to data contamination and adversarial attacks across image and tabular domains.

Datasets

• CIFAR-10 — total ?; splits: train (-1), test (-1)
• FashionMNIST — total ?; splits: train (-1), test (-1)
• Arrhythmia — total ?; splits: train (-1), test (-1)
• Thyroid — total ?; splits: train (-1), test (-1)
• KDD — total ?; splits: train (-1), test (-1)
• KDDRev — total ?; splits: train (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall. Evaluated using a fixed threshold protocol: the threshold is set to classify exactly the number of anomalies present in the test set as anomalies, then standard TP/FP/FN are computed.

Input / output format

Input: Normal data instances (images or tabular vectors) for training; test instances containing both normal and anomalous data.

Output: Anomaly score per instance, used to derive binary predictions via thresholding.

Scoring recipe

# ROC-AUC
roc_auc = roc_auc_score(y_true, anomaly_scores)

# F1 Score (tabular protocol)
n_anomalies = int(np.sum(y_true == 1))
threshold = np.sort(anomaly_scores)[-n_anomalies]
y_pred = (anomaly_scores >= threshold).astype(int)
f1 = f1_score(y_true, y_pred)

Common pitfalls

• Image experiments use a one-class setting: training is performed on a single digit class, not the full CIFAR-10 training set.
• Tabular F1 Score does not use a standard 0.5 threshold; it is fixed to match the exact number of test anomalies to ensure fair comparison across methods.
• Categorical attributes in tabular datasets must be one-hot encoded prior to model input and evaluation.

Evidence (verbatim from paper)

Table 3: Anomaly Detection Accuracy (%) ... F1 Score ... Following the protocol in Zong et al. (2018), the decision threshold value is chosen to result in the correct number of anomalies e.g. if the test set contains Na anomalies, the threshold is selected so that the highest Na scoring examples are classified as anomalies. True positives and negatives are evaluated in the usual way.

Citation

@misc{bergman2020goad,
  title={Classification-Based Anomaly Detection for General Data},
  author={Bergman et al. (2020)},
  year={2020},
  note={arXiv:2005.02359}
}

• arXiv: 2005.02359