Conformal Anomaly Detection Eval

conformal-anomaly-detection-eval

Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors — Oliver Hennhöfer et al. (2024) (arXiv:2402.16388, 2024)

What this evaluates

Evaluates the statistical validity (False Discovery Rate control) and detection sensitivity (statistical power) of cross-conformal anomaly detection methods against split-conformal baselines across datasets of varying sizes and dimensionalities.

Datasets

• ADBench — total ?; splits: (unstated); repo https://github.com/OliverHennhoefer/cross-conformal-anomaly-detection

Metrics

• False Discovery Rate (FDR) (primary) — range: [0, 1]
  • Computed as the False Discovery Proportion (FDP), which is the ratio of inliers incorrectly predicted as outliers to the total number of predicted outliers. Evaluated as the mean and 90th percentile across L=100 test sets, then averaged over J=100 training subsets. Marginal FDR is controlled at α=0.2 using the Benjamini-Hochberg procedure.
• Statistical Power — range: [0, 1]
  • Defined as the proportion of total outliers in a test set that are correctly identified as outliers. Reported as the mean and 90th percentile across L=100 test sets, then averaged over J=100 training subsets.

Input / output format

Input: Feature vectors from benchmark datasets, partitioned into training/calibration sets (containing only inliers) and test sets (containing 90% inliers and 10% outliers).

Output: Binary anomaly labels (outlier/inlier) or anomaly scores used to compute False Discovery Proportion and detection power.

Scoring recipe

for each test set D_test:
  fp = count(inliers predicted as outliers)
  tp = count(outliers predicted as outliers)
  fd = fp + tp
  FDP = fp / fd if fd > 0 else 0
  Power = tp / total_outliers_in_test

# Average over L=100 test sets for a given training set D_j:
cFDR_Dj = mean(FDP over L test sets)
cPower_Dj = mean(Power over L test sets)

# Marginal metrics over J=100 training sets:
mFDR = mean(cFDR_Dj over J sets)
mPower = mean(cPower_Dj over J sets)

# Apply Benjamini-Hochberg at alpha=0.2 for FDR control

Common pitfalls

• Test sets are sampled randomly and may overlap between each other, which can inflate variance estimates if not accounted for.
• Jackknife variants become computationally prohibitive on medium/large datasets, forcing researchers to switch to CV variants with fixed k=20, which changes the calibration set size dynamics.
• FDR control relies on the Benjamini-Hochberg procedure at a nominal alpha=0.2; applying it incorrectly or ignoring the marginal vs conditional distinction breaks the statistical guarantees.

Evidence (verbatim from paper)

Derived methods, and the split-conformal approach for reference, were applied to ten benchmark datasets as found in ADBench... For the evaluation, we are interested in the FDR conditional on D_j defined as the expectation value... with FDP(D^test; D_j) as the false discovery proportion of inliers in the test set that was incorrectly reported as outliers... and the statistical power... where Power(D^test; D_j) is defined as the proportion of total outliers in D^test correctly identified as outliers.

Citation

@misc{hennhofer2024leaveoneout,
  title={Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors},
  author={Oliver Hennhöfer et al. (2024)},
  year={2024},
  note={arXiv:2402.16388}
}

• arXiv: 2402.16388