Nab Eval

nab-eval

Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark — Lavin et al. (2015) (arXiv:1510.03336, 2015)

What this evaluates

Evaluates real-time anomaly detection algorithms on streaming time-series data, measuring their ability to detect natural and synthetic anomalies while penalizing false alarms and delayed detections.

Datasets

• NAB 1.0 — total 58; splits: (unstated); repo https://github.com/numenta/NAB

Metrics

• NAB Score (primary) — range: other
  • A composite score calculated per application profile (Standard, Reward low FP, Reward low FN) that rewards early true positives and penalizes false positives and false negatives. Scores are normalized to a standard range, with higher values indicating better real-time detection performance.

Input / output format

Input: Streaming time-series data files containing labeled natural and synthetic anomalies.

Output: Per-timestep anomaly scores or binary anomaly labels produced by the detector.

Scoring recipe

def compute_nab_score(predictions, gold, profile):
    early_reward = calculate_early_detection_bonus(predictions, gold)
    fp_penalty = calculate_false_positive_penalty(predictions, gold)
    fn_penalty = calculate_false_negative_penalty(predictions, gold)
    if profile == 'Reward low FP':
        score = early_reward - 2 * fp_penalty
    elif profile == 'Reward low FN':
        score = early_reward - 0.5 * fn_penalty
    else:
        score = early_reward - fp_penalty - fn_penalty
    return normalize_to_standard_range(score)

Common pitfalls

• Algorithms' parameters were optimized per profile to yield the best possible NAB scores, which may overstate real-world generalization.
• Random detection baselines do not score near zero due to the NAB score optimization step, making chance performance non-intuitive.
• Scores are reported separately for three application profiles (Standard, Reward low FP, Reward low FN) rather than a single aggregate metric.

Evidence (verbatim from paper)

Table 1 summarizes the scores for all algorithms on the three application profiles using the data files in NAB 1.0. The HTM detector achieves the best overall scores, followed by Etsy and Twitter. Although the HTM detector performs markedly better, the Etsy and Twitter algorithms perform significantly better than chance across all three application profiles.

Citation

@misc{lavin2015nab,
  title={Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark},
  author={Lavin et al. (2015)},
  year={2015},
  note={arXiv:1510.03336}
}

• arXiv: 1510.03336