Matbench Eval

matbench-eval

Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm — Dunn et al. (2020) (arXiv:2005.00707, 2020)

What this evaluates

Evaluates machine learning models on predicting materials properties (e.g., elastic moduli, band gaps, formation energies) from crystal structures or compositions. It probes generalization across diverse data sizes, input types, and property domains using a standardized, pre-cleaned suite of 13 supervised tasks.

Datasets

• Matbench test suite v0.1 — total ?; splits: nested cross-validation (-1); repo https://github.com/hackingmaterials/automatminer

Metrics

• error estimation (RMSE/Accuracy) (primary) — range: other
  • Task-dependent: Root Mean Squared Error (RMSE) for regression tasks; Accuracy or F1-score for binary classification tasks. Computed over nested cross-validation folds to mitigate selection bias.

Input / output format

Input: Single material primitive per sample: either chemical composition or composition plus crystal structure.

Output: Target property value: continuous scalar for regression tasks; binary label for classification tasks.

Scoring recipe

# Nested cross-validation for error estimation
outer_folds = KFold(n_splits=K, shuffle=True)
inner_folds = KFold(n_splits=J, shuffle=True)
errors = []
for train_idx, test_idx in outer_folds.split(data):
    X_train, y_train = data[train_idx], targets[train_idx]
    X_test, y_test = data[test_idx], targets[test_idx]
    best_model = tune_hyperparams(X_train, y_train, inner_folds)
    preds = best_model.predict(X_test)
    err = compute_metric(y_test, preds) # RMSE or Accuracy
    errors.append(err)
final_score = mean(errors)

Common pitfalls

• Using raw, unpreprocessed datasets instead of the provided precleaned versions, which breaks homogeneity and invalidates fair comparison.
• Applying standard train/val/test splits instead of nested cross-validation, introducing model and sample selection bias.
• Mixing tasks or datasets without maintaining the prescribed input/output formats, leading to unfair comparisons across different property domains.

Evidence (verbatim from paper)

To mitigate model and sample selection biases, each task uses a consistent nested cross-validation procedure for error estimation (see Methods). The number of samples in each task ranges from 312 to 132,752... Each task is a self-contained dataset containing a single material primitive as input (either composition or composition plus crystal structure) and target property as output for each sample.

Citation

@misc{dunn2020matbench,
  title={Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm},
  author={Dunn et al. (2020)},
  year={2020},
  note={arXiv:2005.00707}
}

• arXiv: 2005.00707