Dti Benchmark Eval

dti-benchmark-eval

FlexMol: A Flexible Toolkit for Benchmarking Molecular Relational Learning — Liu et al. (2024) (arXiv:2410.15010, 2024)

What this evaluates

Evaluates the ability of molecular models to predict drug-target interactions (DTI) by classifying whether a given drug and protein target pair binds. It probes the model's capacity to integrate diverse molecular representations (sequences, graphs, structures) and interaction layers to distinguish positive binding pairs from negative ones.

Datasets

• Davis — total ?; splits: train (-1), val (-1), test (-1)
• BIOSNAP — total 13741; splits: train (-1), val (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve, measuring the trade-off between precision and recall across thresholds, particularly sensitive to class imbalance.

Input / output format

Input: Paired drug and protein molecular data, which can include drug sequences (ds), drug graphs (dg), protein sequences (ps), and protein graphs (pg) depending on the encoder configuration.

Output: Binary interaction probability or score indicating the likelihood of binding between the drug and protein pair.

Scoring recipe

def compute_metrics(y_true, y_pred):
    roc_auc = roc_auc_score(y_true, y_pred)
    pr_auc = average_precision_score(y_true, y_pred)
    return {'roc-auc': roc_auc, 'pr-auc': pr_auc}
# y_true: 1 for positive (Kd<30 or known binding), 0 for negative (sampled unseen pairs)
# Results are averaged over 5 random 7:2:1 train/val/test splits.

Common pitfalls

• Negative samples are generated by sampling unseen pairs rather than random negative sampling, which may inflate performance compared to standard benchmarks.
• A random 7:2:1 train/val/test split is used instead of target-independent or scaffold splits, potentially causing data leakage and overly optimistic generalization estimates.

Evidence (verbatim from paper)

For both the DAVIS and BIOSNAP datasets, we conducted a random split in the ratio of 7:2:1 for training, validation, and testing, respectively. Each test was repeated five times to mitigate any randomness, and the average results were computed. Table 4 presents the performance metrics of the selected baseline models and the FlexMol models, including ROC-AUC (Receiver Operating Characteristic - Area Under the Curve) and PR-AUC (Precision-Recall Area Under the Curve).

Citation

@misc{liu2024flexmol,
  title={FlexMol: A Flexible Toolkit for Benchmarking Molecular Relational Learning},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.15010}
}

• arXiv: 2410.15010