qhjqhj00 modnet-eval-protocol
Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet — De Breuck et al. (2021) (arXiv:2102.02263, 2021)
What this evaluates
Evaluates the predictive accuracy and uncertainty quantification of a descriptor-based neural network on materials property prediction tasks, with a focus on low-data regimes and bias-imbalance.
Datasets
- Naccarato et al. refractive index — total 4040; splits: train (-1), test (-1)
- Petretto et al. vibrational thermodynamics — total 1265; splits: train (-1), test (-1)
Metrics
mean absolute error(primary) — range: other- Average of absolute differences between predicted and true property values across the test set.
standard deviation— range: other- Standard deviation of predictions across the 125-model ensemble, used as an epistemic uncertainty measure.
Input / output format
Input: Material composition or crystal structure processed into chemical, physical, and geometrical descriptors using Matminer featurizers.
Output: Scalar prediction (ensemble mean) and uncertainty estimate (ensemble standard deviation). Predictions falling outside the training target range padded by 25% are remapped to a uniform distribution within that range.
Scoring recipe
predictions = [model_i(x) for model_i in ensemble]
mean_pred = np.mean(predictions)
uncertainty = np.std(predictions)
r = y_max - y_min
if mean_pred < y_min - 0.25*r or mean_pred > y_max + 0.25*r:
mean_pred = np.random.uniform(y_min, y_max)
mae = np.mean(np.abs(mean_pred - y_true))
Common pitfalls
- Hyperparameters must be optimized via nested cross-validation (internal validation folds), never directly on the test set to avoid overfitting.
- Predictions must be checked against the training target range (padded by 25%); out-of-range values are replaced with random draws to ensure physical plausibility.
Evidence (verbatim from paper)
MODNet has been trained and tested on both single and multi-property tasks with excellent performance compared to contemporary methods for datasets containing 10,000 samples or fewer. For instance, on the refractive index dataset of Naccaratoet al. (4040 samples), MODNet achieves a mean absolute error of 0.051... The specific hyperparameter optimization process used in this work is described in [B]... automated hyperparameter selection ensures a correct assessment of the generalisation error and minimises overfitting. That is, hyperparameters should never be chosen to minimise a test error, but rather chosen internally with a validation procedure.
Citation
@misc{debreuck2021modnet,
title={Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet},
author={De Breuck et al. (2021)},
year={2021},
note={arXiv:2102.02263}
}
- arXiv: 2102.02263