Anomaly Detection Meta Eval

anomaly-detection-meta-eval

A Meta-Analysis of the Anomaly Detection Problem — Emmott et al. (2015) (arXiv:1503.01158, 2015)

What this evaluates

Evaluates anomaly detection algorithms on a large corpus of synthetic datasets systematically varied along four dimensions: point difficulty, semantic variation, relative frequency, and feature relevance. It probes algorithm robustness, generalization across diverse anomaly-generating processes, and the impact of experimental design choices on reported performance.

Datasets

• Synthetic Anomaly Detection Corpus — total 1000; splits: test (1000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. It measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance, integrating the true positive rate against the false positive rate across all thresholds.
• AP — range: [0, 1]
  • Average Precision. It computes the area under the precision-recall curve, summarizing the trade-off between precision and recall across all classification thresholds, particularly sensitive to class imbalance.

Input / output format

Input: Feature vectors for data points, comprising normal instances drawn from a real-world generating process and anomalous instances drawn from a distinct process, with controlled variations in difficulty, frequency, clustering, and feature relevance.

Output: Continuous anomaly scores or binary anomaly labels for each data point.

Scoring recipe

def compute_auc(y_true, y_scores):
    desc_idx = np.argsort(y_scores)[::-1]
    y_true_sorted = y_true[desc_idx]
    tpr = np.cumsum(y_true_sorted) / np.sum(y_true)
    fpr = np.cumsum(1 - y_true_sorted) / np.sum(1 - y_true)
    return np.trapz(tpr, fpr)

def compute_ap(y_true, y_scores):
    desc_idx = np.argsort(y_scores)[::-1]
    y_true_sorted = y_true[desc_idx]
    precision = np.cumsum(y_true_sorted) / np.arange(1, len(y_true_sorted) + 1)
    return np.sum(precision * y_true_sorted) / np.sum(y_true)

Common pitfalls

• Using too few datasets causes algorithms to overfit to specific problem instances rather than generalizing.
• Treating points merely in the tails of the normal distribution as anomalies ignores the requirement that anomalies come from a distinct generating process.
• Ignoring feature relevance leads to high-dimensional spaces where density estimation fails and normal points fall into distribution tails.

Evidence (verbatim from paper)

Using AUC and AP as primary metrics, it reveals significant performance variability across algorithms and demonstrates how experimental design heavily influences results; statistical hypothesis testing shows that many reported improvements are not statistically significant, while a trivial baseline (e.g., uniform scoring) exposes the lack of meaningful performance gains in most cases, highlighting systemic flaws in current anomaly detection evaluation practices.

Citation

@misc{emmott2015meta,
  title={A Meta-Analysis of the Anomaly Detection Problem},
  author={Emmott et al. (2015)},
  year={2015},
  note={arXiv:1503.01158}
}

• arXiv: 1503.01158