Point Adjust F1

point-adjust-f1

Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology — Sehili et al. (2023) (arXiv:2308.13068, 2023)

What this evaluates

Evaluates anomaly detection performance by granting full credit for all points in an anomalous segment if at least one point is detected, often inflating scores for algorithms that merely hit a segment once.

Datasets

• (no dataset; pure metric skill)

Metrics

• point-adjust F1 (primary) — range: [0, 1]
  • F1_pa = 2 * P_pa * R_pa / (P_pa + R_pa), where R_pa = 1 if at least one point in an anomalous segment is detected, else 0. P_pa = TP_pa / (TP_pa + FP), with TP_pa equal to the total number of points in any segment where at least one point was detected.

Input / output format

Input: Multivariate time series data with ground-truth binary anomaly labels per point.

Output: Binary anomaly prediction mask per point, or continuous anomaly scores thresholded to produce a binary mask.

Scoring recipe

def point_adjust_f1(preds, gold, segments):
    tp_pa = 0
    fp = 0
    for seg in segments:
        if np.any(preds[seg] == 1):
            tp_pa += len(seg)
        fp += np.sum(preds == 1)
    r_pa = 1.0 if tp_pa > 0 else 0.0
    p_pa = tp_pa / (tp_pa + fp) if (tp_pa + fp) > 0 else 0.0
    f1_pa = 2 * p_pa * r_pa / (p_pa + r_pa) if (p_pa + r_pa) > 0 else 0.0
    return f1_pa

Common pitfalls

• Detecting a single point in a long anomalous segment grants full credit for all points in that segment, artificially inflating recall and F1.
• Random guessing can achieve high scores if the contamination rate is low and segments are long.
• F1 score is highly sensitive to contamination rate and segment length under this protocol, making cross-dataset comparisons unreliable.

Evidence (verbatim from paper)

The F1 score is calculated in terms of precision P and recall R as follows: F1 = 2PR/(P+R) for P = TP/(TP+FP) and R = TP/(TP+FN)

Citation

@misc{sehili2023mvts,
  title={Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology},
  author={Sehili et al. (2023)},
  year={2023},
  note={arXiv:2308.13068}
}

• arXiv: 2308.13068