Pdbbind Binding Affinity Eval

pdbbind-binding-affinity-eval

Improved prediction of ligand-protein binding affinities by meta-modeling — Lee et al. (2023) (arXiv:2310.03946, 2023)

What this evaluates

This benchmark evaluates the ability of docking tools, deep learning models, and meta-modeling ensembles to predict ligand-protein binding affinities. It probes how well different feature representations (physical scores, sequence-based DL outputs, physicochemical properties) generalize to unseen protein-ligand complexes.

Datasets

• PDBbind — total ?; splits: train (-1), val (-1), test (266)

Metrics

• Pearson correlation coefficient (primary) — range: [-1, 1]
  • Measures the linear correlation between predicted binding affinities and experimental reference values. Calculated as the covariance of the two variables divided by the product of their standard deviations.

Input / output format

Input: Protein-ligand complex identifiers or sequences/structures, along with derived features such as docking scores (SMINA, Vinardo), deep learning model predictions, molecular weight, and PCA-transformed feature vectors.

Output: A single predicted binding affinity value per complex.

Scoring recipe

def calculate_pearson_r(predictions, gold):
    n = len(predictions)
    mean_pred = sum(predictions) / n
    mean_gold = sum(gold) / n
    cov = sum((p - mean_pred) * (g - mean_gold) for p, g in zip(predictions, gold))
    std_pred = (sum((p - mean_pred)**2 for p in predictions) / n) ** 0.5
    std_gold = (sum((g - mean_gold)**2 for g in gold) / n) ** 0.5
    return cov / (std_pred * std_gold)

Common pitfalls

• Data leakage from training set overlap: The CoreSet benchmark overlaps with PDBbind 2013, which was used to train the Vinardo scoring function, potentially inflating test performance.
• Training/validation split dependency: Some meta-models are trained on the exact same RefinedSet\CoreSet used to fine-tune the underlying DL models, which can artificially inflate training correlations and requires strict external validation on the CoreSet to assess true generalization.
• RMSD filtering artifact: Applying strict RMSD cutoffs (e.g., < 3 Å) drastically reduces dataset size, which can spuriously alter correlation metrics unless validated with subsampling simulations.

Evidence (verbatim from paper)

We compare SMINA and Vinardo scores against the experimental binding affinities using both RMSD filter types for the two scoring functions and cutoffs of 101 Angstroms, 100 Angstroms and 3 Angstroms (Figure 2). The results indicate that the outputs of the docking tools consistently have a moderate correlation with the experimental results (on the order of 0.5), for both the RefinedSet\CoreSet, and the CoreSet. The model average Pearson correlation coefficients were 0.468, 0.475, and 0.746 for the CoreSet, respectively (Figure 3A; D1, D2, and D3).

Citation

@misc{lee2023improved,
  title={Improved prediction of ligand-protein binding affinities by meta-modeling},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2310.03946}
}

• arXiv: 2310.03946