Cold Dti Eval

cold-dti-eval

Attending on Multilevel Structure of Proteins enables Accurate Prediction of Cold-Start Drug-Target Interactions — Zhang et al. (2025) (arXiv:2510.04126, 2025)

What this evaluates

Evaluates a model's ability to predict drug-target binding interactions under cold-start conditions where either drugs, proteins, or both are completely unseen during training. It probes the model's capacity to generalize across different protein structural granularities (primary to quaternary) and handle severe class imbalance inherent in biological interaction datasets.

Datasets

• DrugBank — total ?; splits: train (-1), val (-1), test (-1)
• BindingDB — total ?; splits: train (-1), val (-1), test (-1)
• BioSNAP — total ?; splits: train (-1), val (-1), test (-1)
• Human — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the overall discriminating ability of the model across all classification thresholds.
• AUPR — range: [0, 1]
  • Area under the precision-recall curve. Particularly suitable for imbalanced data as it emphasizes the quality of positive predictions.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Balances the trade-off between capturing true positives and avoiding false positives.

Input / output format

Input: Drug representations (molecular graphs or sequences) paired with protein sequences annotated with multi-level structural features (primary, secondary, tertiary, and quaternary structures).

Output: Binary binding probability score indicating whether a drug-target pair interacts (positive) or not (negative).

Scoring recipe

def compute_metrics(y_true, y_pred_proba):
    y_pred = (y_pred_proba >= 0.5).astype(int)
    auc = roc_auc_score(y_true, y_pred_proba)
    aupr = average_precision_score(y_true, y_pred_proba)
    f1 = f1_score(y_true, y_pred)
    return {'AUC': auc, 'AUPR': aupr, 'F1': f1}

Common pitfalls

• DTI datasets are highly imbalanced; reporting overall accuracy is misleading and explicitly discouraged by the authors.
• Cold-start splits enforce strict entity separation: 'cold drug' means zero overlapping drugs across splits, 'cold protein' means zero overlapping proteins, and 'cold pair' means neither overlaps.
• All reported results must be averaged over 3 random runs; single-run evaluations are not comparable to the paper's benchmarks.
• Quaternary protein structure interactions contribute negligibly to performance, as ablation studies show near-identical results when removed.

Evidence (verbatim from paper)

DTI datasets are typically imbalanced, with far fewer positive interactions than negative ones. In such cases, overall accuracy can be misleading, as a model that simply predicts the majority class would still achieve a high score. Therefore, we focus on metrics that better reflect the ability to correctly identify both positive and negative classes. Specifically, we adopt the AUC, the AUPR and the F1 score. AUC measures the overall discriminating ability across thresholds, AUPR is particularly suitable for imbalanced data as it emphasizes the quality of positive predictions, and F1, as the harmonic mean of precision and recall, balances the trade-off between capturing true positives and avoiding false positives.

Citation

@misc{zhang2025coldti,
  title={Attending on Multilevel Structure of Proteins enables Accurate Prediction of Cold-Start Drug-Target Interactions},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.04126}
}

• arXiv: 2510.04126