Pdbbind Low Similarity Eval

pdbbind-low-similarity-eval

Improving generalisability of 3D binding affinity models in low data regimes — Buhmann et al. (2024) (arXiv:2409.12995, 2024)

What this evaluates

Evaluates how well 3D binding affinity models generalise to unseen proteins and novel ligands in low-data regimes. It uses a strict low-Tanimoto-similarity split of the PDBBind dataset to prevent data leakage and benchmark generalisation capabilities.

Datasets

• PDBBind — total ?; splits: train (-1), test (-1); repo https://github.com/Exscientia/low-sim-pdbbind

Metrics

• performance (primary) — range: other
  • Standard regression metric for binding affinity prediction (typically RMSE or MAE between predicted and experimental values). The paper refers to it generally as 'performance'.

Input / output format

Input: 3D coordinates and atom types for protein pockets and ligands, optionally with explicitly added hydrogen atoms.

Output: Continuous binding affinity value (e.g., pKd, pKi, or ΔG).

Scoring recipe

def score(predictions, gold):
    rmse = np.sqrt(np.mean((predictions - gold) ** 2))
    mae = np.mean(np.abs(predictions - gold))
    return {'rmse': rmse, 'mae': mae}

Common pitfalls

• Data leakage if train/test splits are based on high structural similarity rather than low Tanimoto similarity.
• Ignoring hydrogen atoms in protein structures, which significantly hurts performance in low-data regimes.
• Using single-protein local models that fail to generalise to unseen proteins compared to global 3D models.

Evidence (verbatim from paper)

The results reveal a clear advantage for global 3D models over local models in low data regimes. ... As seen in Figure 3, the performance of the global 3D models is relatively consistent with the level of training data (improving slightly) whereas the Single-Protein local model suffers greatly at low data, dramatically improving with increasing amounts of training data.

Citation

@misc{buhmann2024lowsimpdbbind,
  title={Improving generalisability of 3D binding affinity models in low data regimes},
  author={Buhmann et al. (2024)},
  year={2024},
  note={arXiv:2409.12995}
}

• arXiv: 2409.12995