Drugood Eval

drugood-eval

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations — Ji et al. (2022) (arXiv:2201.09637, 2022)

What this evaluates

Evaluates the out-of-distribution (OOD) generalization and robustness of graph neural networks and sequence models on molecular binding affinity prediction tasks under various domain shifts and annotation noise levels.

Datasets

• DrugOOD — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic Curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.

Input / output format

Input: Molecular graphs (derived from SMILES) for ligand-binding affinity prediction (LBAP), and protein amino acid sequences for structure-based affinity prediction (SBAP).

Output: Predicted binding affinity scores or classification labels, generated via a readout function and MLP layer on top of the backbone encoder.

Scoring recipe

def compute_auroc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

Common pitfalls

• Using an in-distribution validation set for early stopping and hyperparameter tuning instead of the specified OOD validation set.
• Assuming domain generalization algorithms will consistently outperform Empirical Risk Minimization (ERM) on molecular graph data, as the paper shows ERM often remains competitive or superior.
• Failing to respect the specific domain split definitions (assay, scaffold, size, protein, protein-family), which dictate the nature of the distribution shift being evaluated.

Evidence (verbatim from paper)

We adopt the Area under the ROC Curve (AUROC) to estimate model performance; the higher score is better. In all tables in this paper, we report in parentheses the standard deviation of 3 replications, which measures the variability among replications. All datasets show performance drops due to distribution shift, with substantially better ID performance than OOD performance.

Citation

@misc{ji2022drugood,
  title={DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations},
  author={Ji et al. (2022)},
  year={2022},
  note={arXiv:2201.09637}
}

• arXiv: 2201.09637