Orionbench Eval

orionbench-eval

OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User — Alnegheimish et al. (2023) (arXiv:2310.17748, 2023)

What this evaluates

Evaluates unsupervised time series anomaly detection pipelines across diverse real-world and synthetic datasets. It measures detection accuracy for both point and segment anomalies while tracking computational efficiency and model stability over continuous benchmarking cycles.

Datasets

• OrionBench (NASA, NAB, Yahoo S5, UCR) — total 742; splits: test (-1); repo https://github.com/sintel-dev/Orion

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed over predicted anomaly labels versus ground truth labels.
• Precision — range: [0, 1]
  • Ratio of correctly predicted anomalies to all predicted anomalies.
• Recall — range: [0, 1]
  • Ratio of correctly predicted anomalies to all actual anomalies.
• Runtime — range: seconds
  • Total elapsed time in seconds to train the pipeline and run inference on a single signal.

Input / output format

Input: Univariate or multivariate time series signals with corresponding ground truth anomaly labels.

Output: Per-timestep anomaly scores or binary anomaly flags indicating detected anomalies.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    tp = sum(p == 1 and g == 1 for p, g in zip(predictions, ground_truth))
    fp = sum(p == 1 and g == 0 for p, g in zip(predictions, ground_truth))
    fn = sum(p == 0 and g == 1 for p, g in zip(predictions, ground_truth))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Reconstruction-based models (e.g., LSTM AE, TadGAN) consistently underperform on point anomalies due to smoothed anomaly scores.
• Hyperparameter choices, particularly window size, significantly impact results; static windows may not generalize across signals of varying lengths.
• Cloud-based services (e.g., Azure AD) tend to over-flag anomalies, artificially inflating F1 on high-anomaly datasets while degrading on sparse ones.

Evidence (verbatim from paper)

The output result is stored as a detailed .csv file that shows for each pipeline and signals performance metrics such as accuracy, precision, recall, and F1 score.

Citation

@misc{alnegheimish2023orionbench,
  title={OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User},
  author={Alnegheimish et al. (2023)},
  year={2023},
  note={arXiv:2310.17748}
}

• arXiv: 2310.17748