Dti Regression Eval

dti-regression-eval

Multi-View Self-Attention for Interpretable Drug-Target Interaction Prediction — Agyemang et al. (2020) (arXiv:2005.00397, 2020)

What this evaluates

Evaluates a model's ability to predict continuous binding affinity for drug-target pairs across different cold-start and warm-start scenarios. It probes the model's generalization to unseen drugs, unseen targets, and fully seen interactions using regression metrics.

Datasets

• Davis — total 31824; splits: warm (-1), cold-drug (-1), cold-target (-1)
• Metz — total 35259; splits: warm (-1), cold-drug (-1), cold-target (-1)
• KIBA — total 160296; splits: warm (-1), cold-drug (-1), cold-target (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)). Lower is better.
• R^2 — range: other
  • Coefficient of determination (referred to as Pearson correlation coefficient in the paper): 1 - SS_res/SS_tot. Higher is better.
• CI — range: [0, 1]
  • Concordance Index: proportion of concordant pairs among all comparable pairs. Higher is better.

Input / output format

Input: Drug-target pair represented by compound features (e.g., ECFP8, GraphConv) and target features (e.g., RNN-PSC, PSC).

Output: Continuous binding affinity value (regression output).

Scoring recipe

def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
    concordant = discordant = tied = 0
    for i in range(len(y_true)):
        for j in range(i + 1, len(y_true)):
            if y_true[i] != y_true[j]:
                if (y_pred[i] - y_pred[j]) * (y_true[i] - y_true[j]) > 0:
                    concordant += 1
                elif (y_pred[i] - y_pred[j]) * (y_true[i] - y_true[j]) < 0:
                    discordant += 1
                else:
                    tied += 1
    ci = (concordant + 0.5 * tied) / (concordant + discordant + tied)
    return rmse, r2, ci

Common pitfalls

• Cold-drug and cold-target splits require strict separation of drugs/targets between train and test sets to avoid data leakage.
• Metrics are averaged across 5-fold CV and multiple random seeds, not just a single split.
• CI is calculated on test set predictions, often requiring pairwise comparison of predicted affinities.

Evidence (verbatim from paper)

As regards evaluation metrics, we measured the Root Mean Squared Error (RMSE) and Pearson correlation coefficient ($R^{2}$) on the test set in each CV-fold. Additionally, we measured the Concordance Index (CI) on the test set, as proposed by[[23]]. We followed the averaging CV approach, where the reported metrics are the averages across the different folds.

Citation

@misc{agyemang2020multiview,
  title={Multi-View Self-Attention for Interpretable Drug-Target Interaction Prediction},
  author={Agyemang et al. (2020)},
  year={2020},
  note={arXiv:2005.00397}
}

• arXiv: 2005.00397