M2hub Eval

m2hub-eval

M$^2$Hub: Unlocking the Potential of Machine Learning for Materials Discovery — Du et al. (2023) (arXiv:2307.05378, 2023)

What this evaluates

Evaluates graph neural networks on predicting material properties from 3D crystal and molecular structures. It probes model performance across diverse material types, physical properties, and realistic data partitioning strategies.

Datasets

• OMDB — total ?; splits: random (-1), composition (-1), system (-1)
• QMOF — total ?; splits: random (-1), composition (-1), system (-1)
• MP — total ?; splits: random (-1), composition (-1), system (-1)
• ISMETAL — total ?; splits: random (-1), composition (-1), system (-1)
• EDOS — total ?; splits: random (-1), composition (-1), system (-1)
• PDOS — total ?; splits: random (-1), composition (-1), system (-1)
• DF2D — total ?; splits: random (-1), composition (-1), system (-1)
• Perovskites — total ?; splits: random (-1), composition (-1), system (-1)
• EFORM — total ?; splits: random (-1), composition (-1), system (-1)
• Phonons — total ?; splits: random (-1), composition (-1), system (-1)
• Dielectric — total ?; splits: random (-1), composition (-1), system (-1)
• LOG_GVRH — total ?; splits: random (-1), composition (-1), system (-1)
• LOG_KVRH — total ?; splits: random (-1), composition (-1), system (-1)
• OC20 — total ?; splits: random (-1), composition (-1), system (-1)
• QM9 — total ?; splits: random (-1), composition (-1), system (-1)

Metrics

• MAE / Accuracy (primary) — range: other
  • Mean Absolute Error (MAE) for regression tasks (e.g., bandgap, formation energy, density of states); classification accuracy for ISMETAL. Lower MAE and higher accuracy indicate better performance.

Input / output format

Input: 3D atomic structures represented as graphs with node features (atom types) and edge features (pairwise distances, lattice vectors).

Output: Continuous property value for regression tasks; binary class label for classification tasks.

Scoring recipe

def compute_metric(predictions, targets, task_type):
    if task_type == 'classification':
        return accuracy_score(targets, predictions)
    else:
        return mean_absolute_error(targets, predictions)

Common pitfalls

• System and composition splits prevent data leakage by separating entire systems or chemical compositions, making evaluation significantly harder than random splits.
• Some tasks have extremely small test sets after splitting, leading to missing results that should be noted.
• Efficiency varies drastically across models; runtime should be reported alongside accuracy for fair comparison.

Evidence (verbatim from paper)

We test all the methods on a list of 13 representative tasks from our benchmarks with three data splits (random, composition and system). ... Benchmark on materials property prediction tasks (different colors denote distinct property types: purple (electrical), yellow (stability), green (thermal), red (optical), blue (mechanical)... Table 3 reports the MAE / Accuracy for each task.)

Citation

@misc{du2023m2hub,
  title={M$^2$Hub: Unlocking the Potential of Machine Learning for Materials Discovery},
  author={Du et al. (2023)},
  year={2023},
  note={arXiv:2307.05378}
}

• arXiv: 2307.05378