Molecule Net Scaffold Eval

molecule-net-scaffold-eval

Multi-View Graph Neural Networks for Molecular Property Prediction — Ma et al. (2020) (arXiv:2005.13607, 2020)

What this evaluates

Evaluates graph neural networks on molecular property prediction tasks, testing the model's ability to capture multi-view (node and edge) structural information for accurate classification and regression of chemical properties.

Datasets

• MoleculeNet (11 datasets) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate for binary classification tasks.
• MAE — range: other
  • Mean Absolute Error, calculated as the average of absolute differences between predicted and true regression values.
• RMSE — range: other
  • Root Mean Squared Error, calculated as the square root of the average of squared differences between predicted and true regression values.

Input / output format

Input: Molecular graphs represented as node features (atoms) and edge features (bonds).

Output: Binary classification labels for classification tasks; continuous scalar values for regression tasks.

Scoring recipe

def compute_auc_roc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def compute_mae(y_true, y_pred):
    return mean(abs(y_true - y_pred))

def compute_rmse(y_true, y_pred):
    return sqrt(mean((y_true - y_pred) ** 2))

Common pitfalls

• Uses scaffold splitting instead of random splitting, which is more challenging and practical for molecular data.
• Regression task results are deferred to the appendix and not reported in the main text.
• Evaluates robustness by reporting mean and standard deviation over 10 runs with different random seeds.

Evidence (verbatim from paper)

Dataset Splitting. We apply the scaffold splitting for all tasks on all datasets, which is more practical and challenging than random splitting. More details about this splitting method is introduced in Appendix E.1. Evaluation Metrics. All classification task are evaluated by AUC-ROC. For the regression task, we apply MAE and RMSE to evaluate the performance of regression task on different datasets.

Citation

@misc{ma2020multiview,
  title={Multi-View Graph Neural Networks for Molecular Property Prediction},
  author={Ma et al. (2020)},
  year={2020},
  note={arXiv:2005.13607}
}

• arXiv: 2005.13607