Shap Diversity Ensemble Eval

shap-diversity-ensemble-eval

Analyzing Shapley Additive Explanations to Understand Anomaly Detection Algorithm Behaviors and Their Complementarity — Levy et al. (2026) (arXiv:2602.00208, 2026)

What this evaluates

Probes how well SHAP-based feature attribution divergence correlates with prediction/output diversity across unsupervised anomaly detection algorithms, and quantifies how this explanation-driven diversity impacts the robustness and accuracy of model ensembles.

Datasets

• ADBench subset (16 datasets: anthyroid, breastw, glass, Hepatitis, Lymphography, mammography, PageBlocks, Pima, Stamps, thyroid, vertebral, vowels, WBC, Wilt, wine, yeast) — total ?; splits: train (-1), test (-1)

Metrics

• AUCPR (primary) — range: [0, 1]
  • Area Under the Precision-Recall Curve, computed on binary predictions versus ground truth labels to handle class imbalance.
• ρPS — range: [-1, 1]
  • Pearson linear correlation between SHAP value vectors of two models across test instances.
• ρNDCG — range: [0, 1]
  • NDCG similarity between SHAP feature importance rankings of two models.
• ρScores — range: [-1, 1]
  • Pearson linear correlation between anomaly score vectors of two models.
• J — range: [0, 1]
  • Jaccard index between binary anomaly prediction sets of two models.
• Mantel correlation (r_M) — range: [-1, 1]
  • Statistical correlation between two distance/similarity matrices, assessed via permutation test.

Input / output format

Input: Raw feature vectors (≤20 features) from tabular datasets. Each instance is passed to 14 UAD models to generate anomaly scores and binary predictions, and to Kernel SHAP to compute feature attributions.

Output: Per instance: anomaly score, binary prediction (normal/anomalous), and SHAP feature attribution values. Ensembles output aggregated anomaly scores via rank aggregation, yielding a final AUCPR score on the test set.

Scoring recipe

1. Compute SHAP values per instance using Kernel SHAP (k=50 k-means background).
2. Compute similarity matrices across models:
   rho_PS = pearson_corr(shap_m1, shap_m2)
   rho_NDCG = ndcg_similarity(rank(shap_m1), rank(shap_m2))
   rho_Scores = pearson_corr(scores_m1, scores_m2)
   J = jaccard_index(pred_m1, pred_m2)
3. For ensembles (size 3), apply rank aggregation to anomaly scores.
4. Evaluate ensemble performance: aucpr = compute_aucpr(y_true, ensemble_scores)
5. Assess matrix correlation: r_M = mantel_test(matrix_A, matrix_B, n_perm=999)
6. Predict ensemble gain via linear regression on mean individual AUCPR and diversity (1 - rho_PS).

Common pitfalls

• Default anomaly thresholds are used for all models, which may not be optimal for imbalanced datasets and can skew Jaccard/score correlations.
• SHAP approximations via k-means background summarization (k=50) may introduce attribution noise, affecting ρPS and ρNDCG stability.
• Diversity is not universally beneficial; negative correlations with AUCPR occur when a single dominant model exists or datasets are too complex, requiring careful model quality filtering.

Evidence (verbatim from paper)

We computed the four similarity matrices ρPS (linear correlation between SHAP values), ρNDCG (similarity of SHAP feature importance rankings), ρScores (linear correlation between anomaly scores), J (jaccard index between anomaly predictions) for each dataset.

Citation

@misc{levy2026shap,
  title={Analyzing Shapley Additive Explanations to Understand Anomaly Detection Algorithm Behaviors and Their Complementarity},
  author={Levy et al. (2026)},
  year={2026},
  note={arXiv:2602.00208}
}

• arXiv: 2602.00208