Soc Dgl Dti Eval

soc-dgl-dti-eval

SOC-DGL: Social Interaction Behavior Inspired Dual Graph Learning Framework for Drug-Target Interaction Identification — Zhao et al. (2025) (arXiv:2506.01405, 2025)

What this evaluates

This evaluation protocol assesses a model's capability to predict binary drug-target interactions (DTI) using graph-based representations. It specifically probes performance under both balanced and highly imbalanced data distributions, as well as generalization to unseen drugs or targets in cold-start scenarios.

Datasets

• KIBA — total ?; splits: 10-fold cross-validation (-1)
• Davis — total ?; splits: 10-fold cross-validation (-1)
• BindingDB — total ?; splits: 10-fold cross-validation (-1)
• DrugBank — total ?; splits: 10-fold cross-validation (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• AUPR — range: [0, 1]
  • Area under the precision-recall curve, emphasizing performance on the positive class, particularly useful under class imbalance.
• F1_score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall).
• ACC — range: [0, 1]
  • Accuracy: (True Positives + True Negatives) / Total Samples.
• Recall — range: [0, 1]
  • True Positive Rate: True Positives / (True Positives + False Negatives).
• Precision — range: [0, 1]
  • Positive Predictive Value: True Positives / (True Positives + False Positives).
• Specificity — range: [0, 1]
  • True Negative Rate: True Negatives / (True Negatives + False Positives).

Input / output format

Input: Drug and target molecular representations processed through dual graph learning modules (ADGL and EDGL) to form drug-target pairs for interaction prediction.

Output: Binary classification probability or discrete label (positive/negative) indicating whether a drug-target pair interacts.

Scoring recipe

def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    tp = np.sum((y_pred == 1) & (y_true == 1))
    tn = np.sum((y_pred == 0) & (y_true == 0))
    fp = np.sum((y_pred == 1) & (y_true == 0))
    fn = np.sum((y_pred == 0) & (y_true == 1))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    acc = (tp + tn) / len(y_true)
    specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    auroc = roc_auc_score(y_true, y_pred_proba)
    auprc = average_precision_score(y_true, y_pred_proba)
    return {'AUROC': auroc, 'AUPR': auprc, 'F1_score': f1, 'ACC': acc, 'Recall': recall, 'Precision': precision, 'Specificity': specificity}

Common pitfalls

• Evaluating on imbalanced datasets (1:10 positive:negative ratio) without using imbalance-aware loss functions or adjusting decision thresholds will severely skew Precision and Recall metrics.
• Cold-start experiments require strict isolation of unseen drugs or targets from the training graph; standard random splits will leak structural information and artificially inflate AUROC/AUPR.
• 10-fold cross-validation must be applied consistently across all baseline methods; reporting single-split results breaks comparability with the paper's statistical validation (paired t-tests + Fisher's Combined Probability Test).

Evidence (verbatim from paper)

To minimize data variability, 10-fold cross-validation was employed to evaluate model performance. During each fold, the test set was masked, and the performance is evaluated using metrics such as AUROC, AUPR, F1_score, ACC, Recall and Precision.

Citation

@misc{zhao2025socdgl,
  title={SOC-DGL: Social Interaction Behavior Inspired Dual Graph Learning Framework for Drug-Target Interaction Identification},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2506.01405}
}

• arXiv: 2506.01405