Hilad Anomaly Discovery Eval

hilad-anomaly-discovery-eval

Effectiveness of Tree-based Ensembles for Anomaly Discovery: Insights, Batch and Streaming Active Learning — Das et al. (2019) (arXiv:1901.08930, 2019)

What this evaluates

Evaluates the effectiveness of tree-based ensemble anomaly detectors in human-in-the-loop active learning settings. It measures how quickly an algorithm can discover anomalies by querying a limited budget of instances, comparing batch and streaming data paradigms.

Datasets

• Abalone — total 1920; splits: pool (1920)
• ANN-Thyroid-1v3 — total 3251; splits: pool (3251)
• Cardiotocography — total 1700; splits: pool (1700)
• KDD-Cup-99 — total 63009; splits: pool (63009)
• Mammography — total 11183; splits: pool (11183)
• Shuttle — total 12345; splits: pool (12345)
• Yeast — total 1191; splits: pool (1191)
• Covtype — total 286048; splits: pool (286048)
• Electricity — total 27447; splits: pool (27447)
• Weather — total 13117; splits: pool (13117)

Metrics

• anomaly_discovery_rate (primary) — range: percent
  • Percentage of total anomalies discovered relative to the total number of anomalies in the dataset, plotted against the cumulative number of queried instances. Calculated as (discovered_anomalies / total_anomalies) * 100.

Input / output format

Input: Feature vectors from benchmark datasets, provided either as a static unlabeled pool (batch setting) or sequentially in streaming windows of size K.

Output: Anomaly scores for each instance, used to rank candidates and select the top-B instances for human labeling per query round.

Scoring recipe

def compute_discovery_curve(discovered_set, total_anomalies, query_budget):
    rates = []
    for q in range(1, query_budget + 1):
        discovered_so_far = len(discovered_set[:q])
        rate = (discovered_so_far / total_anomalies) * 100
        rates.append(rate)
    return rates

Common pitfalls

• Budget B is a fixed number of queries (e.g., 300 for batch), not a percentage of the dataset size.
• Streaming experiments require strict adherence to window size K and queries-per-window Q (e.g., Q=20).
• Drift adaptation uses a KL-divergence 95th percentile threshold to replace trees, not a fixed replacement rate.
• Electricity and Weather datasets had their anomaly classes down-sampled to exactly 5% prior to evaluation.

Evidence (verbatim from paper)

For each variant of the human-in-the-loop framework HiLAD, we plot the percentage of the total number of anomalies shown to the analyst versus the number of instances queried; this is the most relevant metric for an analyst in any real-world application.

Citation

@misc{das2019effectiveness,
  title={Effectiveness of Tree-based Ensembles for Anomaly Discovery: Insights, Batch and Streaming Active Learning},
  author={Das et al. (2019)},
  year={2019},
  note={arXiv:1901.08930}
}

• arXiv: 1901.08930