Gnn Explainer Eval

gnn-explainer-eval

Explaining the Explainers in Graph Neural Networks: a Comparative Study — Longa et al. (2022) (arXiv:2210.15304, 2022)

What this evaluates

Evaluates the quality and interpretability of explanations generated by various Graph Neural Network (GNN) explainers across different architectures and graph datasets. It probes how well explanations align with human expectations (plausibility) and model decision logic (fidelity).

Datasets

• Grid — total ?; splits: test (-1)
• Grid-House — total ?; splits: test (-1)
• Stars — total ?; splits: test (-1)
• House-Color — total ?; splits: test (-1)

Metrics

• F1-Fidelity (primary) — range: [0, 1]
  • Measures the overlap between the explanation mask and the model's actual decision logic, typically computed as an F1 score between the predicted important subgraph and the model's attention or gradient map.
• Plausibility — range: [0, 1]
  • Measures how close the explanation is to human-expected ground truth or intuitive concepts, often evaluated via human studies or predefined structural benchmarks.

Input / output format

Input: Graph structures (nodes, edges, features) for node or graph classification tasks.

Output: Explanation masks indicating node or edge importances (continuous weights or binary masks).

Scoring recipe

def evaluate(explainer, model, dataset):
    mask = explainer.explain(model, dataset)
    fidelity = compute_f1_fidelity(mask, model_predictions)
    plausibility = compute_plausibility(mask, human_ground_truth)
    aggregated = aggregate_scores(fidelity, plausibility)
    return aggregated

Common pitfalls

• Overreliance on consistency metrics without considering structural fidelity.
• Node-based explainers may outperform edge-based ones on node classification but underperform on graph classification.
• Aggregating scores across diverse datasets can mask dataset-specific explainability challenges.

Evidence (verbatim from paper)

With this question we would like to understand which is the architecture that achieves the best score, either in terms of f1-fidelity or plausibility.

Citation

@misc{longa2022explaining,
  title={Explaining the Explainers in Graph Neural Networks: a Comparative Study},
  author={Longa et al. (2022)},
  year={2022},
  note={arXiv:2210.15304}
}

• arXiv: 2210.15304