Glassbench Eval

glassbench-eval

Roadmap on machine learning glassy dynamics — Jung et al. (2023) (arXiv:2311.14752, 2023)

What this evaluates

Evaluates machine learning models' ability to predict particle-level dynamic propensity and dynamic heterogeneity from static amorphous structural configurations in glass-forming liquids.

Datasets

• GlassBench — total ?; splits: train (-1), test (-1)

Metrics

• Pearson correlation coefficient ($\rho_P$) (primary) — range: other
  • Computed over the entire dataset between MD ground truth propensities $\mathcal{R}_i^{\text{MD}}$ and ML predictions $\mathcal{X}_i^{\text{ML}}$: $\rho_P = \text{cov}(\mathcal{R}^{\text{MD}}, \mathcal{X}^{\text{ML}}) / \sqrt{\text{var}(\mathcal{R}^{\text{MD}})\text{var}(\mathcal{X}^{\text{ML}})}$. Must be calculated over all particles of the same type together, not averaged per structure or across mixed types.

Input / output format

Input: Static particle configurations (positions, types) for Kob-Andersen (3D) or KA2D (2D) mixtures, optionally preprocessed into inherent or cage states.

Output: Per-particle dynamic propensity score $\mathcal{X}_i^{\text{ML}}$ (or bond-breaking propensity) for each configuration.

Scoring recipe

import numpy as np
def compute_pearson(md_labels, ml_predictions):
    # Compute over entire dataset, not per-structure or across types
    cov = np.cov(md_labels, ml_predictions)[0, 1]
    var_md = np.var(md_labels, ddof=1)
    var_ml = np.var(ml_predictions, ddof=1)
    return cov / np.sqrt(var_md * var_ml)

Common pitfalls

• Calculating Pearson correlation per structure and then averaging yields systematically different (higher) results.
• Averaging over particles of different types significantly inflates the correlation and should be avoided.
• Relying solely on Pearson correlation is insufficient; models may fail to capture dynamic heterogeneity ($\chi_4$) despite high correlation.
• Correlation coefficients exhibit system dependence (e.g., higher for hard spheres or icosahedral systems than Kob-Andersen mixtures).

Evidence (verbatim from paper)

A common metric used to assess the performance of different techniques is the Pearson correlation coefficient, $\rho_{P}=\frac{\text{cov}(\mathcal{R}^{\text{MD}}{i},\mathcal{X}{i}^{\text{ML}})}{\sqrt{\text{var}(\mathcal{R}^{\text{MD}}{i})\text{var}(\mathcal{X}{i}^{\text{ML}})}}$, between the labels $\mathcal{R}{i}^{\text{MD}}$ for each particle $i$ of type 1 in the entire dataset as obtained from molecular dynamics (MD) simulations, and the ML output $\mathcal{X}{i}^{\text{ML}}$. Calculating the Pearson correlation for each structure individually and then averaging yields slightly different results, and Pearson correlations appear to be systematically higher. Similarly, calculating Pearson correlation over particles of different type significantly increases the correlation and this should be avoided.

Citation

@misc{jung2023roadmap,
  title={Roadmap on machine learning glassy dynamics},
  author={Jung et al. (2023)},
  year={2023},
  note={arXiv:2311.14752}
}

• arXiv: 2311.14752