Bxaic Eval

bxaic-eval

B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data — Proszewska et al. (2025) (arXiv:2505.22252, 2025)

What this evaluates

Evaluates the faithfulness and localization accuracy of explainable AI (XAI) methods for Graph Neural Networks on molecular graphs. It measures how well explainers identify ground-truth chemical motifs (nodes/edges) versus correctly identifying when the entire graph is important, using threshold-free metrics.

Datasets

• B-XAIC — total 50000; splits: test (-1); HF mproszewska/B-XAIC; repo https://github.com/mproszewska/B-XAIC

Metrics

• NE (primary) — range: [0, 1]
  • Node Explanation score measuring the precision/localization of highlighted nodes against ground-truth chemical motifs. Computed as the overlap between explainer-attributed nodes and ground-truth substructures, averaged across all 7 chemical tasks.
• SE — range: [0, 1]
  • Subgraph Explanation score measuring the recall/completeness of the highlighted subgraph against ground-truth motifs. Computed as the coverage of ground-truth substructures by the explainer's output, averaged across tasks.

Input / output format

Input: Molecular graphs represented as node features, edge features, and adjacency matrices for GNN processing.

Output: Soft masks or continuous importance scores assigned to each node and/or edge by the explainer method.

Scoring recipe

def compute_ne_se(explainer_scores, ground_truth_mask, top_k=10):
    predicted_mask = select_top_k(explainer_scores, top_k)
    tp = sum(predicted_mask & ground_truth_mask)
    ne = tp / sum(predicted_mask) if sum(predicted_mask) > 0 else 0
    se = tp / sum(ground_truth_mask) if sum(ground_truth_mask) > 0 else 0
    return ne, se
# Average NE and SE across all 7 chemical tasks and all test instances

Common pitfalls

• Using a fixed threshold to binarize explainer scores across different methods, as attribution scales vary widely (e.g., near-zero vs. ~4.5).
• Assuming high predictive F1 scores guarantee faithful explanations; GIN achieves >98% F1 but explainers still fail to highlight simple patterns.
• Evaluating node and edge explanations interchangeably, as some explainers only support one modality.

Evidence (verbatim from paper)

Gradient-based methods are, on average, better at localizing important patterns than other methods. However, they tend to highlight molecular fragments even when the pattern is absent, resulting in low NE scores.

Citation

@misc{proszewska2025bxaic,
  title={B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data},
  author={Proszewska et al. (2025)},
  year={2025},
  note={arXiv:2505.22252}
}

• arXiv: 2505.22252