Dta Affinity Prediction Eval

dta-affinity-prediction-eval

HGTDP-DTA: Hybrid Graph-Transformer with Dynamic Prompt for Drug-Target Binding Affinity Prediction — Xi Xiao et al. (2024) (arXiv:2406.17697, 2024)

What this evaluates

Evaluates a model's ability to predict the binding affinity between small molecule drugs and protein targets. It probes regression accuracy, ranking consistency, and correlation strength on standardized drug-target interaction datasets.

Datasets

• Davis — total 30056; splits: train (25746), test (5010)
• KIBA — total 118254; splits: train (98545), test (19709)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted and ground-truth affinity values. Lower is better.
• CI — range: [0, 1]
  • Concordance Index measuring the probability that the model correctly ranks a randomly selected pair of drug-target interactions by affinity. Higher is better.
• r2_m — range: [0, 1]
  • Improved R-squared metric for external prediction, calculated as R^2 * (1 - sqrt(R^2_0 / R^2)), where R^2_0 is R^2 computed without an intercept. Higher is better.
• Pearson — range: [-1, 1]
  • Pearson correlation coefficient between predicted and actual binding affinities. Higher is better.

Input / output format

Input: Drug SMILES strings and target protein sequences, processed into molecular graphs and sequence embeddings respectively.

Output: A single continuous floating-point value representing the predicted binding affinity (pKd for Davis, KIBA score for KIBA).

Scoring recipe

def compute_metrics(preds, golds):
    mse = np.mean((preds - golds) ** 2)
    pearson = np.corrcoef(preds, golds)[0, 1]
    pairs = list(itertools.combinations(range(len(preds)), 2))
    concordant = sum(1 for i, j in pairs if golds[i] != golds[j] and (preds[i]-preds[j])*(golds[i]-golds[j]) > 0)
    total = sum(1 for i, j in pairs if golds[i] != golds[j])
    ci = concordant / total if total > 0 else 0.0
    r2 = 1 - np.sum((preds - golds)**2) / np.sum((golds - np.mean(golds))**2)
    slope = np.sum(preds * golds) / np.sum(preds**2)
    r2_0 = 1 - np.sum((golds - slope * preds)**2) / np.sum((golds - np.mean(golds))**2)
    r2_m = r2 * (1 - np.sqrt(r2_0 / r2)) if r2 > 0 else 0.0
    return {'MSE': mse, 'CI': ci, 'r2_m': r2_m, 'Pearson': pearson}

Common pitfalls

• The train/test splits are fixed per dataset; averaging over 5 random seeds applies only to model training initialization, not to the data split itself.
• The r2_m metric requires calculating R^2_0 without an intercept; using standard R^2 will yield incorrect values.
• CI calculation must correctly exclude pairs with tied ground-truth affinities, as they do not contribute to the concordant count.

Evidence (verbatim from paper)

Specifically, these metrics include the Mean Squared Error (MSE), Concordance Index (CI), $r^{2}_{m}$, and Pearson’s Correlation (Pearson).

Citation

@misc{xiao2024hgtdpdta,
  title={HGTDP-DTA: Hybrid Graph-Transformer with Dynamic Prompt for Drug-Target Binding Affinity Prediction},
  author={Xi Xiao et al. (2024)},
  year={2024},
  note={arXiv:2406.17697}
}

• arXiv: 2406.17697