Molecular Ue Eval

molecular-ue-eval

Uncertainty Estimation for Molecules: Desiderata and Methods — Wollschläger et al. (2023) (arXiv:2306.14916, 2023)

What this evaluates

This benchmark evaluates uncertainty estimation methods for molecular force fields by measuring predictive accuracy on equilibrium structures, calibration of uncertainty scores, and out-of-distribution detection capabilities on non-equilibrium or left-out molecular configurations.

Datasets

• MD17 — total ?; splits: train (-1), test (-1)
• QM7X — total ?; splits: train (-1), test (-1)

Metrics

• MAE — range: other
  • Mean Absolute Error computed separately for energy and forces. Formula: (1/N) * sum(|y_pred - y_true|). Units vary by dataset (kcal/mol or eV for energy; kcal/(mol·Å) or eV/Å for forces).
• Calibration Score — range: other
  • Regression calibration score from Charpentier et al. (2022) measuring how well predicted uncertainties align with actual prediction errors.
• AUC-ROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve computed on uncertainty scores to separate in-distribution from out-of-distribution samples. A value closer to 1 indicates better separation.

Input / output format

Input: Molecular structures represented as atom types and 3D coordinates (equilibrium or non-equilibrium configurations).

Output: Predicted total energy, atomic forces, and associated uncertainty estimates (e.g., variance or predictive log-likelihood).

Scoring recipe

def compute_metrics(predictions, gold, uncertainty_scores, labels):
    # MAE for energy and forces
    mae_energy = np.mean(np.abs(predictions['energy'] - gold['energy']))
    mae_force = np.mean(np.abs(predictions['force'] - gold['force']))
    # AUC-ROC for OOD detection
    auc = roc_auc_score(labels, uncertainty_scores)
    # Calibration score (Charpentier et al. 2022)
    cal = compute_calibration_score(predictions, gold, uncertainty_scores)
    return {'MAE_energy': mae_energy, 'MAE_force': mae_force, 'AUC-ROC': auc, 'Calibration': cal}

Common pitfalls

• MCD dropout sacrifices predictive accuracy for uncertainty estimates, with energy accuracy deteriorating significantly at higher dropout rates.
• Calibration scores can be misleading if prediction accuracy is poor, as uniform distributions minimize the score regardless of predictive quality.
• Locality (size consistency) is only satisfied in expectation for MCD due to its stochastic nature, unlike deterministic methods.

Evidence (verbatim from paper)

For OOD detection we use the area under the receiver operating characteristic curve (AUC-ROC). For each backbone, we use the respective hyperparameters from their papers, listed in Section A.2.

Citation

@misc{wollschlaeger2023uncertainty,
  title={Uncertainty Estimation for Molecules: Desiderata and Methods},
  author={Wollschläger et al. (2023)},
  year={2023},
  note={arXiv:2306.14916}
}

• arXiv: 2306.14916