Dta Coldstart Eval

dta-coldstart-eval

Mitigating cold start problems in drug-target affinity prediction with interaction knowledge transferring — Nguyen et al. (2022) (arXiv:2202.01195, 2022)

What this evaluates

Evaluates drug-target affinity prediction models in cold-start settings (cold-drug and cold-target) to assess generalization to novel drugs or targets using transferred inter-molecular interaction knowledge.

Datasets

• Davis — total ?; splits: val (-1), test (-1)
• Kiba — total ?; splits: val (-1), test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)). Measures average prediction error magnitude.
• Pearson — range: other
  • Pearson correlation coefficient: covariance(x,y) / (std(x)*std(y)). Measures linear correlation between predicted and true affinities.
• Spearman — range: other
  • Spearman rank correlation: Pearson correlation computed on the ranks of y_true and y_pred. Measures monotonic relationship.
• CI — range: [0, 1]
  • Concordance Index: fraction of all concordant pairs among comparable pairs. Measures ranking accuracy of predicted vs true affinities.

Input / output format

Input: Drug representation (SMILES sequence or molecular graph) and protein sequence.

Output: Continuous binding affinity value.

Scoring recipe

import numpy as np
from scipy.stats import pearsonr, spearmanr

def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    pearson, _ = pearsonr(y_true, y_pred)
    spearman, _ = spearmanr(y_true, y_pred)
    n = len(y_true)
    concordant = 0
    total = 0
    for i in range(n):
        for j in range(i + 1, n):
            diff_true = y_true[i] - y_true[j]
            diff_pred = y_pred[i] - y_pred[j]
            if diff_true != 0:
                total += 1
                if diff_true * diff_pred > 0:
                    concordant += 1
                elif diff_true * diff_pred == 0:
                    concordant += 0.5
    ci = concordant / total if total > 0 else 0.0
    return {'RMSE': rmse, 'Pearson': pearson, 'Spearman': spearman, 'CI': ci}

Common pitfalls

• Cold-start evaluation splits (cold-drug vs cold-target) are mentioned but not explicitly defined or provided in the text, complicating exact replication.
• Train/validation/test split sizes and ratios for Davis and Kiba are not stated, requiring external sources or standard splits.
• Multiple metrics are reported without a single headline metric, so the primary evaluation criterion must be inferred (typically RMSE or CI in DTA).

Evidence (verbatim from paper)

We evaluate the model performance on the test set using Root Mean Squared Error (RMSE), Pearson (Benesty et al., 2009), Spearman (Zwillinger and Kokoska, 1999), and Concordance Index (CI) (Gönen and Heller, 2005).

Citation

@misc{nguyen2022mitigating,
  title={Mitigating cold start problems in drug-target affinity prediction with interaction knowledge transferring},
  author={Nguyen et al. (2022)},
  year={2022},
  note={arXiv:2202.01195}
}

• arXiv: 2202.01195