Exathlon Eval

exathlon-eval

Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series — Jacob et al. (2020) (arXiv:2010.05073, 2020)

What this evaluates

Evaluates explainable anomaly detection (AD) and explanation discovery (ED) capabilities on high-dimensional multivariate time series. It probes an algorithm's ability to detect range-based anomalies across four progressive difficulty levels and assesses the quality of generated explanations based on conciseness, consistency, and predictive accuracy.

Datasets

• Exathlon — total ?; splits: train (-1), test (-1); repo https://github.com/exathlonbenchmark/exathlon

Metrics

• Range-based Precision (primary) — range: [0, 1]
  • Extends classical precision to time intervals: TP ranges / (TP ranges + FP ranges). Values are scaled based on AD levels (AD1-AD4) to account for overlap size, detection latency, and duplicate penalties.
• Range-based Recall — range: [0, 1]
  • Extends classical recall to time intervals: TP ranges / (TP ranges + FN ranges). Monotonically decreases across AD levels AD1 to AD4 as requirements for range coverage, early detection, and exactly-once reporting tighten.
• Conciseness — range: other
  • Number of features used in the explanation. For local explanations (ED1), it is the size of the feature set |F_{t,w}|. For global explanations (ED2), it is the average size across the explanation set.
• Consistency — range: other
  • Entropy of the normalized frequency distribution of features across a set of explanations: H(A) = -Σ p(a_j) log2 p(a_j), where p(a_j) is the normalized frequency of feature a_j in the duplicate-preserving union of explanation feature sets.
• Accuracy — range: [0, 1]
  • Point-based precision, recall, and F-score computed by treating the explanation as a binary predictive model (0/1 output) and evaluating it on a held-out test set of anomalous and normal data points.

Input / output format

Input: High-dimensional multivariate time series traces (1000s of dimensions) with ground-truth root cause intervals (RCI) and extended effect intervals (EEI). For AD, models receive normal data for training and mixed normal/anomalous data for testing. For ED, models receive detected anomaly subsequences and corresponding feature vectors.

Output: For AD: binary classification labels (normal vs. anomalous) with associated time intervals. For ED: human-readable explanations (logical formulas, decision trees, or feature importance scores) mapping features to anomaly instances.

Scoring recipe

def compute_range_metrics(pred_intervals, true_intervals, ad_level):
    tp, fp, fn = 0, 0, 0
    for p in pred_intervals:
        if any(overlaps(p, t) for t in true_intervals):
            tp += 1
        else:
            fp += 1
    for t in true_intervals:
        if not any(overlaps(t, p) for p in pred_intervals):
            fn += 1
    # Apply AD-level adjustments (AD1-AD4)
    if ad_level >= 2: tp *= overlap_ratio(p, t)  # proportional to size
    if ad_level >= 3: tp *= latency_weight(p, t)  # penalize late detection
    if ad_level >= 4: tp = 0 if count_duplicates(p) > 1 else tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return precision, recall

Common pitfalls

• Treating AD levels (AD1-AD4) as independent tasks instead of a monotonic hierarchy where higher levels strictly build upon and tighten the requirements of lower levels.
• Applying range-based evaluation logic to ED accuracy; the benchmark explicitly requires point-based precision/recall for ED because most explanations lack temporal pattern characterization.
• Ignoring the semi-supervised training constraint; the benchmark expects models trained only on normal data, yet many practitioners evaluate using mixed or fully labeled training sets.

Evidence (verbatim from paper)

To assess how well an AD algorithm can meet these four functionality levels, we use the customizable accuracy evaluation framework for time series (Tatbul et al., 2018). This framework extends the classical precision/recall from point-based data to range-based data, by introducing a set of tunable parameters. By setting the values of these parameters in a particular way and applying the resulting precision/recall formulas to the output of an AD algorithm, one can assess how well that output measures up to the quality expectations represented by those parameter settings.

Citation

@misc{jacob2020exathlon,
  title={Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series},
  author={Jacob et al. (2020)},
  year={2020},
  note={arXiv:2010.05073}
}

• arXiv: 2010.05073