Ncad Time Series Eval

ncad-time-series-eval

Neural Contextual Anomaly Detection for Time Series — Carmona et al. (2021) (arXiv:2107.07702, 2021)

What this evaluates

Evaluates time series anomaly detection models across unsupervised, semi-supervised, and supervised settings. It probes the model's ability to detect point and segment anomalies using window-based contextual representations and outlier exposure techniques.

Datasets

• SMAP — total ?; splits: train (-1), test (-1)
• MSL — total ?; splits: train (-1), test (-1)
• SWaT — total ?; splits: train (-1), val (-1), test (-1)
• SMD — total ?; splits: train (-1), test (-1)
• Yahoo — total ?; splits: train (-1), val (-1), test (-1)
• KPI — total ?; splits: train (-1), val (-1)

Metrics

• F1 score (primary) — range: percent
  • Point-wise F1 score where a predicted anomaly correctly covers a true anomalous segment if at least one time point overlaps. Best threshold is selected to maximize F1 on the test set (or validation set for hyperparameter tuning).

Input / output format

Input: Time series data split into context and suspect windows, optionally augmented with synthetic anomalies during training.

Output: Binary anomaly labels (0/1) for each time step in the suspect window.

Scoring recipe

def compute_f1(preds, gold):
    segments = find_contiguous_segments(gold)
    tp = sum(any(preds[t] for t in seg) for seg in segments)
    fp = sum(1 for t in range(len(preds)) if preds[t] and gold[t] == 0)
    fn = sum(1 for seg in segments if not any(preds[t] for t in seg))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Using standard point-wise F1 without segment expansion penalizes models that detect the onset of an anomaly but miss its tail.
• Threshold selection varies by dataset: the paper optimizes the threshold on the test set for final reporting, but uses the validation set for hyperparameter tuning when labels are available.
• Dataset splits are not uniform; SWaT, Yahoo, and KPI use custom train/val/test splits, while SMAP, MSL, and SMD use pre-defined splits. Mixing these up breaks comparability.

Evidence (verbatim from paper)

point-wise scores are used, but the predicted labels are expanded to mark an entire true anomalous segment as detected correctly if at least one time point was detected by the model. We align our experimental protocol with this body of prior work and report F1 scores computed by choosing the best threshold on the test set.

Citation

@misc{carmona2021ncad,
  title={Neural Contextual Anomaly Detection for Time Series},
  author={Carmona et al. (2021)},
  year={2021},
  note={arXiv:2107.07702}
}

• arXiv: 2107.07702