Ssm Dta Dta Prediction Eval

ssm-dta-dta-prediction-eval

SSM-DTA: Breaking the Barriers of Data Scarcity in Drug-Target Affinity Prediction — Pei et al. (2022) (arXiv:2206.09818, 2022)

What this evaluates

Evaluates a model's ability to predict binding affinity between drug molecules and target proteins. It probes regression accuracy, correlation strength, and ranking consistency across varying data scarcity and generalization settings.

Datasets

• BindingDB — total ?; splits: train (-1), valid (-1), test (-1)
• DAVIS — total ?; splits: train (-1), valid (-1), test (-1)
• KIBA — total ?; splits: train (-1), valid (-1), test (-1)

Metrics

• Concordance Index (CI) (primary) — range: [0, 1]
  • Measures the probability that predicted affinity values for two random pairs are ordered correctly relative to ground truth. CI = (pairs_correct + 0.5 * pairs_tied) / pairs_admissible.
• RMSE — range: other
  • Root mean square error between ground truth affinity scores and predictions. RMSE = sqrt(mean((t - p)^2)).
• MSE — range: other
  • Mean square error between ground truth affinity scores and predictions. MSE = mean((t - p)^2).
• PC — range: other
  • Pearson correlation coefficient measuring linear correlation between ground truth and predictions.
• R^2 — range: other
  • R-squared representing the proportion of variance in the dependent variable predictable from the independent variable. R^2 = 1 - sum((t-p)^2)/sum((t-mean(t))^2).

Input / output format

Input: Paired drug molecule (SMILES string) and target protein (amino acid sequence).

Output: Continuous regression score representing predicted binding affinity.

Scoring recipe

import numpy as np
def compute_ci(t, p):
    pairs_admissible = 0
    pairs_correct = 0
    pairs_tied = 0
    for i in range(len(t)):
        for j in range(i+1, len(t)):
            if t[i] != t[j]:
                pairs_admissible += 1
                if (t[i] > t[j] and p[i] > p[j]) or (t[i] < t[j] and p[i] < p[j]):
                    pairs_correct += 1
                elif p[i] == p[j]:
                    pairs_tied += 1
    return (pairs_correct + 0.5 * pairs_tied) / pairs_admissible

def compute_rmse(t, p):
    return np.sqrt(np.mean((np.array(t) - np.array(p))**2))

Common pitfalls

• BindingDB and DAVIS require log-transforming raw IC50/Ki/Kd values using -log10(x/10^9), whereas KIBA uses a pre-computed aggregated score that should not be transformed.
• Dataset splits differ by dataset: BindingDB uses a 6:1:3 train/valid/test ratio, while DAVIS and KIBA use a 7:1:2 ratio.
• The 'unknown drug' setting on DAVIS requires filtering drugs via substructural k-means outlier detection before evaluation, rather than using a standard random split.

Evidence (verbatim from paper)

Evaluation Metrics. We use (i) mean square error (MSE), (ii) root mean square error (RMSE), (iii) Pearson correlation coefficient (PC) [48], (iv) concordance index (CI) [49]. MSE and RMSE measure the difference between ground truth values and values predicted by the model, (v) R-squared (R^2) [15]. ... Following previous works [5, 14, 18, 15], results on BindingDB dataset are evaluated on RMSE and PC, results on DAVIS and KIBA datasets are evaluated by MSE and CI scores, and results on DAVIS dataset with unknown drug setting are evaluated on MSE and R^2 .

Citation

@misc{pei2022ssmdta,
  title={SSM-DTA: Breaking the Barriers of Data Scarcity in Drug-Target Affinity Prediction},
  author={Pei et al. (2022)},
  year={2022},
  note={arXiv:2206.09818}
}

• arXiv: 2206.09818