Matsciml Eval

matsciml-eval

MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling — Lee et al. (2023) (arXiv:2309.05934, 2023)

What this evaluates

Evaluates graph neural networks and equivariant point cloud networks on solid-state materials modeling tasks, including energy/force prediction, bandgap/fermi level regression, and crystal symmetry classification. Probes single-task, multi-task, and multi-dataset generalization capabilities.

Datasets

• OpenCatalyst (OC-20) — total ?; splits: train (-1), val (-1), test (-1)
• Materials Project (MP) — total ?; splits: train (-1), val (-1), test (-1)
• LiPS — total ?; splits: train (-1), val (-1), test (-1)
• OQMD — total ?; splits: train (-1), val (-1), test (-1)
• NOMAD — total ?; splits: train (-1), val (-1), test (-1)
• CMD — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: (1/N) * Σ(y_true - y_pred)^2. Used for energy, bandgap, and fermi level prediction.
• MAE — range: other
  • Mean Absolute Error: (1/N) * Σ|y_true - y_pred|. Used for force prediction.
• Accuracy — range: [0, 1]
  • Correct predictions divided by total predictions. Used for stability and space group classification.

Input / output format

Input: Crystal structures represented as graphs or point clouds, including atomic coordinates, species, and periodic boundary conditions.

Output: Predicted scalar values (energy, bandgap, fermi level), force vectors, or class labels (stability, space group).

Scoring recipe

def compute_metric(y_true, y_pred, task_type):
    if task_type in ['energy', 'bandgap', 'fermi']:
        return np.mean((y_true - y_pred) ** 2)  # MSE
    elif task_type == 'force':
        return np.mean(np.abs(y_true - y_pred))  # MAE
    elif task_type in ['stability', 'space_group']:
        return np.mean(y_true == y_pred)  # Accuracy

Common pitfalls

• Comparing to OC20 leaderboard test splits vs. public validation splits leads to unfair comparisons.
• Space group classification suffers from severe class imbalance and hierarchical symmetry, making it inherently difficult for all models.
• Multi-data learning for IS2RE worsens performance because it predicts relaxed energy rather than single-frame energy, unlike other datasets.

Evidence (verbatim from paper)

The results from [Table 3] indicate that: Task Metric ... Energy Prediction S2EF MSE 0.826 ... Force Prediction S2EF MAE 0.957 ... Space ACC 29.8

Citation

@misc{lee2023matsciml,
  title={MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2309.05934}
}

• arXiv: 2309.05934