Gtb Dti Eval

gtb-dti-eval

Benchmark on Drug Target Interaction Modeling from a Structure Perspective — Zhang et al. (2024) (arXiv:2407.04055, 2024)

What this evaluates

Evaluates structure-based drug-target interaction (DTI) prediction models on regression (binding affinity) and classification (binding status) tasks across six standard bioinformatics datasets.

Datasets

• DAVIS — total ?; splits: train|val|test (-1)
• KIBA — total ?; splits: train|val|test (-1)
• BindingDB — total ?; splits: train|val|test (-1)
• Human — total ?; splits: train|val|test (-1)
• Cycles — total ?; splits: train|val|test (-1)
• Drugbank — total ?; splits: train|val|test (-1)

Metrics

• PCC (primary) — range: [0, 1]
  • Pearson Correlation Coefficient. Measures the linear correlation between predicted and true binding affinities. Ranges from -1 to 1, where 1 indicates perfect positive correlation.
• MAE — range: other
  • Mean Absolute Error. Computes the average absolute difference between predicted and true values. Lower is better.
• ROC-AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between positive and negative classes across all classification thresholds.
• PR-AUC — range: [0, 1]
  • Area Under the Precision-Recall Curve. Evaluates performance on imbalanced datasets by plotting precision against recall at various thresholds.

Input / output format

Input: Drug molecular structure (e.g., SMILES string or molecular graph) and target protein structure (e.g., amino acid sequence or protein graph).

Output: Continuous binding affinity value (regression) or binary interaction label/probability (classification).

Scoring recipe

def score_regression(y_true, y_pred):
    pcc = pearsonr(y_true, y_pred)[0]
    mae = mean_absolute_error(y_true, y_pred)
    return {'PCC': pcc, 'MAE': mae}

def score_classification(y_true, y_pred):
    roc_auc = roc_auc_score(y_true, y_pred)
    pr_auc = average_precision_score(y_true, y_pred)
    return {'ROC-AUC': roc_auc, 'PR-AUC': pr_auc}

Common pitfalls

• Datasets lack standardized train/val/test splits across literature, causing unfair comparisons between models.
• Regression metrics like MAE/MSE are scale-dependent, while correlation metrics (PCC/Spearman) are scale-invariant; results are not directly comparable without matching splits.
• Classification tasks often have highly imbalanced positive/negative samples, making ROC-AUC optimistic; PR-AUC is more informative for imbalanced DTI data.

Evidence (verbatim from paper)

The complete result on the regression task is shown in Table 10, and the complete result on classification task is shown in Table 11. Table headers list metrics including MSE, MAE, R2, PCC, C1, Spearman for regression, and ROC-AUC, PR-AUC, Range-AUC, Acc., Precision, Recall, F1 for classification.

Citation

@misc{zhang2024gtbdti,
  title={Benchmark on Drug Target Interaction Modeling from a Structure Perspective},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2407.04055}
}

• arXiv: 2407.04055