Roar Eval

roar-eval

A Benchmark for Interpretability Methods in Deep Neural Networks — Hooker et al. (2018) (arXiv:1806.10758, 2018)

What this evaluates

This benchmark evaluates the quality of feature importance estimators in deep neural networks by measuring how model performance degrades when ranked important features are removed and the model is retrained. It probes whether an interpretability method correctly identifies pixels that the model actually relies on for prediction.

Datasets

• ImageNet — total ?; splits: train (-1), test (-1)
• Birdsnap — total ?; splits: train (-1), test (-1)
• Food 101 — total ?; splits: train (-1), test (-1)

Metrics

• test accuracy (primary) — range: percent
  • Mean classification accuracy on the modified test set, averaged over 5 independent training runs from random initialization.

Input / output format

Input: Original input images paired with a feature importance ranking/map generated by an interpretability estimator. The evaluation pipeline modifies these images by masking the top-ranked pixels according to a removal fraction t.

Output: A scalar test accuracy score (percentage) representing the retrained model's performance on the modified test set.

Scoring recipe

def compute_roar_metric(estimator, train_imgs, test_imgs, t):
    # 1. Generate importance scores for all images
    scores = estimator.rank_features(train_imgs + test_imgs)
    # 2. Create modified datasets by removing top t% pixels per image
    mod_train = mask_top_pixels(train_imgs, scores, t)
    mod_test = mask_top_pixels(test_imgs, scores, t)
    # 3. Retrain ResNet-50 on modified training data (5 runs)
    models = [train_resnet50(mod_train) for _ in range(5)]
    # 4. Evaluate each model on modified test data and average
    accs = [m.evaluate(mod_test) for m in models]
    return sum(accs) / len(accs)

Common pitfalls

• Evaluating feature importance without retraining the model conflates ranking quality with data degradation, as accuracy drops drastically regardless of the estimator used.
• Relying on base estimators (e.g., GRAD, IG, GB) without ensembling often yields rankings that perform worse than random assignment or simple edge filters.
• Failing to use appropriate control baselines (random assignment and Sobel edge filter) makes it impossible to establish a lower bound for method performance.

Evidence (verbatim from paper)

We independently train 5 ResNet-50 models from random initialization on each of these modified dataset and report test accuracy as the average of these 5 runs.

Citation

@misc{hooker2018roar,
  title={A Benchmark for Interpretability Methods in Deep Neural Networks},
  author={Hooker et al. (2018)},
  year={2018},
  note={arXiv:1806.10758}
}

• arXiv: 1806.10758