Explanation Disagreement Eval

explanation-disagreement-eval

The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective — Krishna et al. (2022) (arXiv:2202.01602, 2022)

What this evaluates

Evaluates the consistency of post-hoc explanation methods by quantifying how much feature attributions differ across algorithms for identical model predictions. It probes whether local explanations are reliable and whether practitioners have principled ways to resolve conflicts when different methods yield conflicting importance scores.

Datasets

• COMPAS — total ?; splits: test (1482)
• German Credit — total ?; splits: test (200)
• News text dataset — total ?; splits: test (7600)
• PASCAL VOC 2012 — total ?; splits: test (1449)

Metrics

• L2 distance of feature attributions (primary) — range: other
  • Euclidean distance between attribution vectors generated by two explanation methods for the same instance. Used for tabular data to measure divergence in feature importance scores.
• Rank Correlation — range: other
  • Correlation coefficient computed on the ranked order of feature importances produced by two explanation methods. Used for tabular and text data.
• Cosine distance between attribution maps — range: other
  • 1 minus the cosine similarity between flattened attribution maps. Used for image data to measure disagreement in spatial saliency patterns.

Input / output format

Input: Input instances (tabular features, text sequences, or images) and their corresponding model predictions.

Output: Feature attribution vectors or saliency maps produced by post-hoc explanation methods (LIME, KernelSHAP, Vanilla Gradients, Integrated Gradients, Gradient*Input, SmoothGRAD).

Scoring recipe

def compute_disagreement(attrib_A, attrib_B, data_type):
    if data_type == 'tabular':
        return np.linalg.norm(attrib_A - attrib_B)  # L2 distance
    elif data_type == 'image':
        cos_sim = np.dot(attrib_A.flatten(), attrib_B.flatten()) / \
                  (np.linalg.norm(attrib_A) * np.linalg.norm(attrib_B))
        return 1 - cos_sim  # Cosine distance
    else:
        return 1 - rank_correlation(attrib_A, attrib_B)  # Rank Correlation

Common pitfalls

• Gradient-based explanation methods cannot be applied to tree-based models (Random Forest, Gradient-Boosted Trees) and must be excluded from those comparisons.
• Perturbation-based and step-based methods require convergence checks (monitoring L2 distance changes across sample sizes/steps) before computing disagreement to ensure stable attributions.
• Top-k feature overlap metrics are explicitly avoided for high-dimensional image data in favor of rank correlation and cosine distance.

Evidence (verbatim from paper)

Hence, we use Rank Correlation and cosine distance between attribution maps generated by a pair of explanation methods as the disagreement metric. Higher cosine distance between attribution maps indicate larger disagreement between explanation methods.

Citation

@misc{krishna2022disagreement,
  title={The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective},
  author={Krishna et al. (2022)},
  year={2022},
  note={arXiv:2202.01602}
}

• arXiv: 2202.01602