Muben Uq Eval

muben-uq-eval

MUBen: Benchmarking the Uncertainty of Molecular Representation Models — Li et al. (2023) (arXiv:2306.10060, 2023)

What this evaluates

This benchmark evaluates the uncertainty quantification (UQ) capabilities of molecular representation models across diverse backbone architectures and input modalities. It probes how accurately models predict molecular properties (binary classification and continuous regression) while simultaneously estimating their own predictive uncertainty under both in-distribution and out-of-distribution conditions. The evaluation specifically tests robustness to molecular scaffold shifts, which better simulates real-world drug discovery and materials design scenarios.

Datasets

• MoleculeNet — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds. Ranges from 0 to 1, where 1 indicates perfect discrimination.
• RMSE (primary) — range: [0, inf)
  • Root-mean-square error. Computed as the square root of the average squared difference between predicted and true regression values. Lower values indicate better regression accuracy.
• MAE — range: [0, inf)
  • Mean absolute error. The average of absolute differences between predicted and true regression values. Provides a robust measure of regression error less sensitive to outliers than RMSE.
• ECE — range: [0, 1]
  • Expected Calibration Error. Measures the weighted average absolute difference between predicted confidence and actual accuracy across probability bins. Lower values indicate better calibrated uncertainty estimates.
• NLL — range: [0, inf)
  • Negative Log Likelihood. For classification, computed as -mean(ylog(p) + (1-y)log(1-p)). For regression, computed as the Gaussian NLL: 0.5log(2pisigma^2) + 0.5((y-μ)^2/sigma^2). Lower is better.
• Brier Score — range: [0, 1]
  • Mean squared difference between predicted probabilities and actual binary outcomes: mean((p - y)^2). Evaluates both calibration and sharpness of probabilistic predictions.
• Regression CE — range: [0, 1]
  • Calibration Error for regression tasks. Quantifies the discrepancy between predicted uncertainty (variance) and actual prediction errors across bins. Lower values indicate better uncertainty calibration.

Input / output format

Input: Molecular structures encoded as SMILES strings, 2D molecular graphs, 3D molecular conformations, or fixed 200-dimensional RDKit features, processed by backbone models (ChemBERTa, GROVER, Uni-Mol, DNN, TorchMD-NET, GIN).

Output: Classification: Sigmoid/Softmax probabilities over binary classes. Regression: Predicted mean (μ̂) and variance (σ̂) parameters, with variance constrained to be positive via SoftPlus activation.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob, y_var):
    # Classification
    roc_auc = compute_roc_auc(y_true, y_prob)
    ece = compute_expected_calibration_error(y_true, y_prob, n_bins=15)
    nll = -mean(y_true * log(y_prob) + (1 - y_true) * log(1 - y_prob))
    brier = mean((y_prob - y_true) ** 2)
    # Regression
    rmse = sqrt(mean((y_true - y_pred) ** 2))
    mae = mean(abs(y_true - y_pred))
    nll_reg = mean(0.5 * log(2 * pi * y_var) + 0.5 * ((y_true - y_pred) ** 2) / y_var)
    ce_reg = compute_calibration_error(y_true, y_pred, y_var)
    # Aggregate over 3 seeds (except ensembles)
    return {'ROC-AUC': roc_auc, 'RMSE': rmse, 'MAE': mae,
            'ECE': ece, 'NLL': nll, 'Brier': brier,
            'NLL_reg': nll_reg, 'CE_reg': ce_reg}

Common pitfalls

• Using random data splitting instead of scaffold splitting, which fails to create the intended out-of-distribution evaluation setup and artificially inflates performance estimates.
• Reporting metrics from a single training run instead of averaging across three random seeds (0, 1, 2), except for Deep Ensembles which correctly aggregates predictions before metric computation.
• Treating deterministic variance outputs (from Gaussian NLL training) as equivalent to proper uncertainty estimates from Bayesian or ensemble methods without accounting for their different calibration properties and optimization objectives.

Evidence (verbatim from paper)

Consistent with recommendations from Wu et al. (2018) and other previous works (Fang et al., 2022; Zhou et al., 2023), we report ROC-AUC (area under the receiver operating characteristic curve) as the metric for classification prediction and RMSE (root-mean-square error) and MAE (mean absolute error) for regression. In quantifying classification uncertainty, we use ECE, NLL, and Brier Score as introduced in § 3. For regression, we compute the Gaussian NLL and regression CE. We compute the metrics for each task individually before calculating their macro average. Each reported metric is the average of 3 individual training-test runs with random seeds 0, 1, and 2.

Citation

@misc{li2023muben,
  title={MUBen: Benchmarking the Uncertainty of Molecular Representation Models},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2306.10060}
}

• arXiv: 2306.10060