Boom Ood Eval

boom-ood-eval

BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models — Antoniuk et al. (2025) (arXiv:2505.01912, 2025)

What this evaluates

Evaluates the out-of-distribution (OOD) generalization of machine learning models for molecular property prediction. It probes whether models trained on in-distribution (ID) molecules can accurately extrapolate to novel chemical spaces, highlighting the disconnect between ID accuracy and OOD robustness.

Datasets

• QM9 — total ?; splits: train (-1), test (-1), ood_test (10000)

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error: sqrt(mean((y_pred - y_true)^2)). Used as the primary metric for both ID and OOD performance across all tasks.
• R² — range: [0, 1]
  • Coefficient of Determination: 1 - (sum((y_true - y_pred)^2) / sum((y_true - mean(y_true))^2)). Standard R² is reported for ID splits, while binned R² is used for OOD splits.

Input / output format

Input: Molecular structures provided as 3D coordinates, graph representations, or SMILES strings.

Output: Continuous numerical values corresponding to specific molecular properties (e.g., heat of formation, density, HOMO/LUMO energies, dipole moment, heat capacity).

Scoring recipe

import numpy as np
def compute_rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))
def compute_r2(y_true, y_pred):
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    return 1 - (ss_res / ss_tot)

Common pitfalls

• ID performance does not reliably predict OOD performance; models can overfit to ID space while failing to extrapolate.
• Autoregressive models often fail on OOD splits due to numerical token generation errors (e.g., predicting '00913' for '0.913'), producing characteristic S-shaped parity plots.
• Standard masked language modeling pretraining improves ID accuracy but can significantly degrade OOD performance (e.g., binned R² drops by 39-53%).

Evidence (verbatim from paper)

The Geoformer achieves the best overall ID performance, achieving the lowest ID RMSE on 3 out of 10 tasks. For OOD prediction, GotenNet achieves top performance on 7 out of 10 tasks, and MACE achieves top performance on 2 out of 10 tasks.

Citation

@misc{antoniuk2025boom,
  title={BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models},
  author={Antoniuk et al. (2025)},
  year={2025},
  note={arXiv:2505.01912}
}

• arXiv: 2505.01912