Bones Eval

bones-eval

BONES: a Benchmark fOr Neural Estimation of Shapley values — Napolitano et al. (2024) (arXiv:2407.16482, 2024)

What this evaluates

Evaluates the accuracy and computational efficiency of neural and traditional Shapley value estimators against ground truth attributions across tabular and image datasets. It measures how well different explainers approximate feature importance and how fast they run.

Datasets

• Monks — total 432; splits: train (302), val (130)
• WBC — total 546; splits: train (436), val (110)
• Census — total 26048; splits: train (20838), val (5210)
• Credit — total 24000; splits: train (19200), val (4800)
• Magic — total 15216; splits: train (12172), val (3044)
• ImageNette — total 11432; splits: train (9469), val (1963)
• Pet — total 6614; splits: train (5879), val (735)

Metrics

• L1 distance (primary) — range: other
  • Sum of absolute differences between estimated and ground truth Shapley values across all features. Lower is better.
• L2 distance — range: other
  • Sum of squared differences between estimated and ground truth Shapley values. Lower is better.
• Kendall correlation — range: [-1, 1]
  • Measures the consistency in feature ranking between estimated and ground truth Shapley values. Used for tabular data only.
• Performance metric P — range: [0, 1]
  • Normalized score across explainers: P = 1 - (d_i - d_min) / (d_max - d_min), where d_i is the distance metric for explainer i. Higher is better.
• Inclusion AUC — range: [0, 1]
  • Area under the curve measuring how model prediction scores increase as top-ranked regions are progressively included. Used for image data.
• Exclusion AUC — range: [0, 1]
  • Area under the curve measuring how model prediction scores decrease as top-ranked regions are progressively removed. Used for image data.

Input / output format

Input: Tabular feature vectors or 224x224 pixel images, paired with a pre-trained black-box classifier (e.g., MLP for tabular, ViT for images).

Output: Per-feature or per-pixel Shapley value attribution scores, plus execution time logs for training and inference.

Scoring recipe

def compute_l1_distance(estimates, gold):
    return np.sum(np.abs(np.array(estimates) - np.array(gold)))

def compute_performance_metric(distances):
    d_min, d_max = min(distances), max(distances)
    if d_max == d_min: return 1.0
    return 1.0 - (np.array(distances) - d_min) / (d_max - d_min)

Common pitfalls

• Confusing neural vs. traditional Shapley estimators; the benchmark treats both but they have different computational profiles and accuracy trade-offs.
• Using raw L1/L2 distance without normalization across datasets, making cross-dataset comparisons invalid without the P metric.
• For image data, forgetting that Inclusion/Exclusion AUC requires progressive masking of regions, not just pixel-wise attribution comparison.

Evidence (verbatim from paper)

BONES natively supports evaluation functions suited to quantify the prediction error made by a SVs estimator against a ground truth. It integrates the L1 and L2 distances. Furthermore, for tabular data only, it also supports the Kendall correlation coefficient, which evaluates the consistency in the SVs feature ranking.

Citation

@misc{napolitano2024bones,
  title={BONES: a Benchmark fOr Neural Estimation of Shapley values},
  author={Napolitano et al. (2024)},
  year={2024},
  note={arXiv:2407.16482}
}

• arXiv: 2407.16482