Devnet Anomaly Detection Eval

devnet-anomaly-detection-eval

Deep Anomaly Detection with Deviation Networks — Pang et al. (2019) (arXiv:1911.08623, 2019)

What this evaluates

Evaluates the capability of anomaly detection models to identify rare or deviant data points using only a small set of labeled anomalies as prior knowledge. It probes data efficiency, robustness to varying anomaly contamination levels in unlabeled training data, and the ability to rank anomalies effectively under severe class imbalance.

Datasets

• donors — total 619326; splits: train (-1), test (-1)
• census — total 299285; splits: train (-1), test (-1)
• fraud — total 284807; splits: train (-1), test (-1)
• celeba — total 202599; splits: train (-1), test (-1)
• backdoor — total 95329; splits: train (-1), test (-1)
• URL — total 89063; splits: train (-1), test (-1)
• campaign — total 41188; splits: train (-1), test (-1)
• news20 — total 10523; splits: train (-1), test (-1)
• thyroid — total 7200; splits: train (-1), test (-1)

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Calculates the area under the curve plotting True Positive Rate against False Positive Rate. A value of 1 indicates perfect ranking, while 0.5 indicates random ranking.
• AUC-PR (primary) — range: [0, 1]
  • Calculates the area under the Precision-Recall curve using the average precision method. It is more suitable for highly imbalanced anomaly detection datasets where performance on the positive class is critical.

Input / output format

Input: Multidimensional feature vectors representing data objects. Training consists of an unlabeled dataset (80% of data) with a small prior set of 30 labeled anomalies sampled from the anomaly class.

Output: A continuous anomaly score for each data object, used to rank objects by likelihood of being anomalous.

Scoring recipe

def compute_auc(y_true, y_scores, curve='roc'):
    if curve == 'roc':
        tpr, fpr, _ = roc_curve(y_true, y_scores)
        return auc(fpr, tpr)
    else:
        precision, recall, _ = precision_recall_curve(y_true, y_scores)
        return auc(recall, precision)
# Results averaged over 10 independent runs

Common pitfalls

• AUC-ROC can be misleadingly optimistic on highly imbalanced anomaly detection datasets because it incorporates performance on the majority normal class.
• High-dimensional datasets (e.g., URL, news20) require dimensionality reduction (sparse random projection to 1000D) for baselines like iForest to run efficiently.
• Training contamination level (anomaly percentage in the unlabeled training set) significantly impacts performance; main experiments fix this at 2%.

Evidence (verbatim from paper)

We use two popular and complementary performance metrics, the Area Under Receiver Operating Characteristic Curve (AUC-ROC) and the Area Under Precision-Recall Curve (AUC-PR), to have a comprehensive evaluation of anomaly detectors. AUC-ROC summarizes the ROC curve of true positives against false positives, while AUC-PR is a summarization of the curve of precision against recall. Specifically, an AUC-ROC value of one indicates the best performance, while a value close to 0.5 indicates a random ranking of the objects.

Citation

@misc{pang2019deepanomaly,
  title={Deep Anomaly Detection with Deviation Networks},
  author={Pang et al. (2019)},
  year={2019},
  note={arXiv:1911.08623}
}

• arXiv: 1911.08623