Molecule Property Prediction Eval

molecule-property-prediction-eval

Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study — Deng et al. (2022) (arXiv:2209.13492, 2022)

What this evaluates

Evaluates molecular property prediction models across classification and regression tasks on chemical datasets. It probes model generalization under different data splits (scaffold vs. random) and tests the impact of representation type (learned vs. fixed descriptors) and evaluation metric choice on reported performance.

Datasets

• MoleculeNet — total ?; splits: scaffold (-1), random (-1)
• Opioids-related — total ?; splits: scaffold (-1), random (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across classification thresholds.
• RMSE (primary) — range: [0, ∞)
  • Root Mean Squared Error. The square root of the average of squared differences between predicted and actual continuous values.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve. Evaluates classifier performance on imbalanced datasets by focusing on positive class precision and recall.
• PPV — range: [0, 1]
  • Positive Predictive Value (Precision). The proportion of true positives among all predicted positives.
• NPV — range: [0, 1]
  • Negative Predictive Value. The proportion of true negatives among all predicted negatives.
• MAE — range: [0, ∞)
  • Mean Absolute Error. The average of absolute differences between predicted and actual values.
• R2 — range: (-∞, 1]
  • Coefficient of Determination. Represents the proportion of variance in the dependent variable predictable from the independent variables.
• Pearson_R — range: [-1, 1]
  • Pearson correlation coefficient. Measures the linear correlation between predicted and actual values.

Input / output format

Input: Molecular structures provided as SMILES strings, molecular graphs, or fixed chemical descriptors (e.g., RDKit2D, PhysChem, MACCS keys, MorganBits, MorganCounts, AtomPairs).

Output: Predicted class labels (active/inactive) for classification tasks, or continuous property values (e.g., pIC50, solubility) for regression tasks.

Scoring recipe

def compute_metrics(preds, gold, task):
    if task == 'classification':
        return {
            'AUROC': roc_auc_score(gold, preds),
            'AUPRC': average_precision_score(gold, preds),
            'PPV': precision_score(gold, preds, pos_label=1),
            'NPV': recall_score(gold, preds, pos_label=0)
        }
    else:
        return {
            'RMSE': sqrt(mean_squared_error(gold, preds)),
            'MAE': mean_absolute_error(gold, preds),
            'R2': r2_score(gold, preds),
            'Pearson_R': pearsonr(gold, preds)[0]
        }
# Average over 30 splits and report mean ± std. Use Mann-Whitney U test for pairwise significance.

Common pitfalls

• Concatenating fixed descriptors to learned representations inflates performance and misleads assessment of representation learning models.
• Choosing different evaluation metrics (e.g., AUROC vs. PPV/NPV, or Pearson_R vs. R2) can lead to disparate conclusions about which model is best.
• Data splitting can be customized to favor a specific model, introducing bias in generalizability claims.
• Scaffold splits create larger gaps in label distributions and structural similarity compared to random splits, making evaluation harder but more realistic.

Evidence (verbatim from paper)

In MoleculeNet, each benchmark dataset comes with a recommended evaluation metric. However, in real-world drug discovery, these metrics may not always be appropriate (see Sec. 2.4.2). In this section, we compared model performance using a variety of evaluation metrics, in addition to the recommended ones. For classification tasks, we calculated AUROC, AUPRC, PPV and NPV (see Sec. 4.2.1). For regression tasks, we calculated RMSE, MAE, R2 and Pearson_R (see Sec. 4.2.2). As shown in Fig. 3b & c, when using the recommended AUROC, RF achieves higher performance than MolBERT, GROVER and GROVER_RDKit in BBBP (p < 0.05).

Citation

@misc{deng2022unraveling,
  title={Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study},
  author={Deng et al. (2022)},
  year={2022},
  note={arXiv:2209.13492}
}

• arXiv: 2209.13492