Dti Prediction Eval

dti-prediction-eval

A Graph-in-Graph Learning Framework for Drug-Target Interaction Prediction — Yuehua Song, Yong Gao (2025) (arXiv:2507.11757, 2025)

What this evaluates

This benchmark evaluates a model's ability to predict drug-target interactions by integrating molecular graphs and protein sequences into a heterogeneous interaction network. It probes the model's capacity to learn hierarchical graph representations and distinguish interacting from non-interacting drug-protein pairs.

Datasets

• DTI Benchmark — total 1923; splits: (unstated)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, plotting True Positive Rate against False Positive Rate across classification thresholds. Higher values indicate better global separability.
• AUPRC — range: [0, 1]
  • Area under the precision-recall curve, evaluating the trade-off between precision and recall across thresholds. Particularly meaningful for sparse interaction datasets.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall, calculated at a fixed 0.5 decision threshold. Reflects the classifier's discriminative power under balanced sampling.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, measuring the correlation between observed and predicted binary classifications. Provides a nuanced view of predictive consistency across both positive and negative samples.

Input / output format

Input: Drug and target nodes represented as molecular graphs (derived from SMILES) and protein sequences, embedded as meta-nodes in a bipartite drug-target interaction graph. Pairs are constructed based on known interaction edges.

Output: A scalar probability score indicating the likelihood of interaction for each drug-target pair.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score, f1_score, matthews_corrcoef

def compute_metrics(y_true, y_pred_prob):
    auc = roc_auc_score(y_true, y_pred_prob)
    auprc = average_precision_score(y_true, y_pred_prob)
    y_pred = (y_pred_prob >= 0.5).astype(int)
    f1 = f1_score(y_true, y_pred)
    mcc = matthews_corrcoef(y_true, y_pred)
    return {'AUC': auc, 'AUPRC': auprc, 'F1': f1, 'MCC': mcc}

Common pitfalls

• Training uses hard negative sampling with a 1:1 positive-to-negative ratio, which can artificially inflate F1 and AUPRC if the test set retains this balance rather than reflecting real-world sparsity.
• The dataset is described as a 'novel benchmark' with no public release link or HuggingFace ID provided, making independent replication difficult.
• AUC evaluates ranking robustness and is less affected by the hard negative sampling strategy, whereas AUPRC and F1 are highly sensitive to the positive class prevalence in the evaluation split.

Evidence (verbatim from paper)

To evaluate the performance of the models, we selected four key metrics: the area under the receiver operating characteristic curve (AUC) [58], the area under the precision-recall curve (AUPRC) [59, 61], the F1 Score [60], and the Matthews Correlation Coefficient (MCC) [62]. These metrics were chosen to provide a comprehensive view of the model's predictive capabilities, especially considering the use of hard negative sampling during the training phase.

Citation

@misc{song2025gig,
  title={A Graph-in-Graph Learning Framework for Drug-Target Interaction Prediction},
  author={Yuehua Song, Yong Gao (2025)},
  year={2025},
  note={arXiv:2507.11757}
}

• arXiv: 2507.11757