Drug Target Interaction Eval

drug-target-interaction-eval

A large dataset curation and benchmark for drug target interaction — Golts et al. (2024) (arXiv:2401.17174, 2024)

What this evaluates

Evaluates computational models on predicting binary drug-target interactions using standardized bioactivity data. It probes the model's ability to learn molecular and protein representations and generalize across different data splits (lenient, cold-ligand, cold-target).

Datasets

• Curated DTI dataset — total 12644133; splits: Lenient (12644133), Cold-ligand (12318559), Cold-target (12318548)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• AUPR — range: [0, 1]
  • Area under the Precision-Recall curve, measuring the trade-off between precision and recall across classification thresholds, particularly informative for imbalanced datasets.

Input / output format

Input: Protein target sequence and ligand SMILES string.

Output: Probability of interaction (scalar between 0 and 1).

Scoring recipe

def compute_metrics(y_true, y_pred):
    from sklearn.metrics import roc_auc_score, average_precision_score
    auroc = roc_auc_score(y_true, y_pred)
    aupr = average_precision_score(y_true, y_pred)
    return {'AUROC': auroc, 'AUPR': aupr}

Common pitfalls

• Dataset is highly imbalanced (~1.1-1.3% positive class), so AUROC can be misleading; AUPR is often more reliable for ranking performance.
• Results are split-dependent: 'Cold-ligand' and 'Cold-target' evaluate out-of-distribution generalization, while 'Lenient' evaluates in-distribution performance. Comparing across splits is invalid.
• Training is truncated to ~250k iterations (24h) on a single GPU, so reported metrics likely understate the model's full convergence potential.

Evidence (verbatim from paper)

Table 4. Benchmark results

Citation

@misc{golts2024drugtarget,
  title={A large dataset curation and benchmark for drug target interaction},
  author={Golts et al. (2024)},
  year={2024},
  note={arXiv:2401.17174}
}

• arXiv: 2401.17174