Molecular Bayesian Eval

molecular-bayesian-eval

A benchmark study on reliable molecular supervised learning via Bayesian learning — Hwang et al. (2020) (arXiv:2006.07021, 2020)

What this evaluates

Evaluates the reliability and predictive performance of Graph Neural Networks (GNNs) trained with Bayesian inference methods on molecular property prediction tasks. It specifically probes how well these models calibrate their uncertainty and generalize to out-of-distribution molecular scaffolds compared to standard maximum a posteriori (MAP) training.

Datasets

• BBBP — total ?; splits: train (-1), val (-1), test (-1)
• BACE — total ?; splits: train (-1), val (-1), test (-1)
• HIV — total ?; splits: train (-1), val (-1), test (-1)
• Tox21 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ECE (primary) — range: percent
  • Expected Calibration Error. Computed by binning predictions by confidence, then calculating the weighted average of the absolute difference between accuracy and mean confidence per bin. Lower values indicate better calibration.
• AUROC — range: percent
  • Area Under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance across all classification thresholds. Higher values indicate better discriminative performance.

Input / output format

Input: Molecular graphs represented as node features (atomic properties) and edge features (bond types/properties) for binary classification tasks.

Output: A scalar predictive probability (between 0 and 1) indicating the likelihood of the molecule belonging to the positive class.

Scoring recipe

def compute_ece(predictions, labels, n_bins=15):
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for i in range(n_bins):
        mask = (predictions >= bin_boundaries[i]) & (predictions < bin_boundaries[i+1])
        if mask.sum() == 0: continue
        bin_acc = labels[mask].mean()
        bin_conf = predictions[mask].mean()
        ece += (mask.sum() / len(labels)) * abs(bin_acc - bin_conf)
    return ece * 100

Common pitfalls

• Using random data splits instead of scaffold-based splits, which fails to properly evaluate out-of-distribution generalization on molecular graphs.
• Interpreting ECE as a performance metric where higher values are better; ECE measures miscalibration, so lower values indicate better reliability.
• Comparing single Bayesian models directly against ensembles without accounting for the computational overhead or the specific ensemble configuration.

Evidence (verbatim from paper)

In Table 1 and 2, we show the prediction results for the four different prediction tasks – BACE, BBBP, HIV, and Tox21 predictions – where GIN is set as the baseline model architecture and various Bayesian learning methods (Ensemble, MC-DO, BBB, SGLD, SWA, and SWAG) are used. We observe that all Bayesian methods are helpful for improving both prediction reliability (lower ECE) and performance (higher AUROC).

Citation

@misc{hwang2020benchmark,
  title={A benchmark study on reliable molecular supervised learning via Bayesian learning},
  author={Hwang et al. (2020)},
  year={2020},
  note={arXiv:2006.07021}
}

• arXiv: 2006.07021