Nab Yahoo Anomaly Detection Eval

nab-yahoo-anomaly-detection-eval

Unsupervised Anomaly Detection in Stream Data with Online Evolving Spiking Neural Networks — Maciąg et al. (2019) (arXiv:1912.08785, 2019)

What this evaluates

Evaluates unsupervised anomaly detection algorithms on streaming time-series data. It probes the model's ability to identify point, contextual, and collective anomalies in highly imbalanced real-world and synthetic datasets without labeled training data.

Datasets

• Numenta Anomaly Benchmark — total 58; splits: test (-1)
• Yahoo Anomaly Dataset — total 367; splits: test (-1)

Metrics

• Precision — range: [0, 1]
  • |TP| / (|TP| + |FP|)
• Recall — range: [0, 1]
  • |TP| / (|TP| + |FN|)
• F-measure (primary) — range: [0, 1]
  • 2 * (Precision * Recall) / (Precision + Recall)
• Balanced Accuracy (BA) — range: [0, 1]
  • 0.5 * (|TP|/(|TP|+|FN|) + |TN|/(|TN|+|FP|))
• Matthews Correlation Coefficient (MCC) — range: [-1, 1]
  • (|TP||TN| - |FP||FN|) / sqrt((|TP|+|FP|)(|TP|+|FN|)(|TN|+|FP|)(|TN|+|FN|))

Input / output format

Input: CSV files containing timestamp and input value time series. Data is processed sequentially as a streaming sequence of scalar values.

Output: Binary classification label per input value indicating whether it is classified as anomalous or non-anomalous.

Scoring recipe

tp = sum(pred == 1 and gold == 1)
fp = sum(pred == 1 and gold == 0)
fn = sum(pred == 0 and gold == 1)
tn = sum(pred == 0 and gold == 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
ba = 0.5 * (recall + (tn / (tn + fp) if (tn + fp) > 0 else 0.0))
mcc = (tp*tn - fp*fn) / math.sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)) if denominator > 0 else 0.0

Common pitfalls

• Labeling inconsistencies in NAB and Yahoo where some true anomalies are unlabeled or normal points are incorrectly labeled, which can artificially lower recall and F-measure.
• Strong class imbalance (<10% anomalies, often <1%) makes standard accuracy misleading; balanced accuracy or MCC should be prioritized for evaluation.
• Grid search for window size (W_size) and anomaly factor (epsilon) is performed per dataset/file, which may lead to overfitting to the evaluation set if not properly separated from tuning.

Evidence (verbatim from paper)

In the experimental phase, we compare anomaly detection quality of our approach to the other state-of-the-art methods and algorithms provided in the literature. To this end, we use five measures of detection quality: precision, recall, F-measure, balanced accuracy (BA) and Matthews correlation coefficient (MCC).

Citation

@misc{maciag2019unsupervised,
  title={Unsupervised Anomaly Detection in Stream Data with Online Evolving Spiking Neural Networks},
  author={Maciąg et al. (2019)},
  year={2019},
  note={arXiv:1912.08785}
}

• arXiv: 1912.08785