Drug Pair Scoring Eval

drug-pair-scoring-eval

ChemicalX: A Deep Learning Library for Drug Pair Scoring — Rozemberczki et al. (2022) (arXiv:2202.05240, 2022)

What this evaluates

This evaluation benchmarks deep learning architectures on predicting drug-drug interactions, polypharmacy side effects, and drug synergy. It measures how well models encode molecular graphs and combine them to score pairwise biological outcomes across multiple pharmacological domains.

Datasets

• TWOSIDES — total 499582; splits: train (-1), test (-1)
• Drugbank DDI — total 383496; splits: train (-1), test (-1)
• DrugComb — total 659333; splits: train (-1), test (-1)
• DrugCombDB — total 191391; splits: train (-1), test (-1)
• OncolyPharm — total 23052; splits: train (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
• AUPR — range: [0, 1]
  • Area under the precision-recall curve. Measures the trade-off between precision and recall across all classification thresholds, particularly sensitive to class imbalance.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall. Computed using a fixed 0.5 probability cutoff on the predicted propensities rather than optimizing the threshold per dataset.

Input / output format

Input: Pairs of molecular graphs (or SMILES strings) optionally paired with a biological context (e.g., cell line), along with a binary label indicating interaction/synergy/polypharmacy effect.

Output: A continuous probability score (propensity) between 0 and 1 representing the likelihood of the drug pair exhibiting the target effect.

Scoring recipe

def compute_metrics(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    auroc = auc(fpr, tpr)
    precision, recall, _ = precision_recall_curve(y_true, y_pred)
    aupr = auc(recall, precision)
    y_pred_binary = (y_pred >= 0.5).astype(int)
    tp = np.sum((y_pred_binary == 1) & (y_true == 1))
    fp = np.sum((y_pred_binary == 1) & (y_true == 0))
    fn = np.sum((y_pred_binary == 0) & (y_true == 1))
    prec = tp / (tp + fp + 1e-8)
    rec = tp / (tp + fn + 1e-8)
    f1 = 2 * prec * rec / (prec + rec + 1e-8)
    return {'AUROC': auroc, 'AUPR': aupr, 'F1': f1}

Common pitfalls

• The F1 score is computed using a fixed 0.5 probability cutoff rather than optimizing the threshold per dataset or model.
• Results are averaged over 10 random 80/20 splits with a fixed seed; performance can vary significantly depending on the specific split composition.
• Datasets differ in context cardinality and drug overlap, making direct cross-dataset comparison of absolute metric values difficult without normalization.

Evidence (verbatim from paper)

Using 80% of the labeled instances we trained deep pair scoring models with the default settings and scored on the remainder. We computed mean predictive performances with standard errors from 10 data splits and reported the AUROC, AUPR, and F1 score values in Table 4. The data splits were seeded to help the comparison of results and the F1 scores used a 0.5 cutoff of propensities output by various architectures.

Citation

@misc{rozemberczki2022chemicalx,
  title={ChemicalX: A Deep Learning Library for Drug Pair Scoring},
  author={Rozemberczki et al. (2022)},
  year={2022},
  note={arXiv:2202.05240}
}

• arXiv: 2202.05240