Casp 2016 Dti Eval

casp-2016-dti-eval

PIGNet: A physics-informed deep learning model toward generalized drug-target interaction predictions — Moon et al. (2020) (arXiv:2008.12249, 2020)

What this evaluates

Evaluates deep learning models for drug-target interaction prediction across four tasks: scoring (affinity correlation), ranking (pose affinity ordering), docking (native pose retrieval from decoys), and screening (true binder identification among random molecules). Probes both regression accuracy and virtual screening generalization.

Datasets

• CASF-2016 — total ?; splits: test (-1); repo https://github.com/jaechanglim/DTI_PDBbind.git
• CSAR NRC-HiQ — total ?; splits: test (-1)

Metrics

• Scoring power (Pearson's R) — range: [0, 1]
  • Pearson correlation coefficient between predicted and experimental binding affinities.
• Ranking power (Spearman's ρ) — range: [0, 1]
  • Spearman's rank-correlation coefficient measuring the ability to correctly rank binding affinities of true binders.
• Docking power (Top-1 Success Rate) (primary) — range: [0, 1]
  • Percentage of complexes where the native binding pose is ranked within the top 1 candidates among computationally generated decoys.
• Screening power (Top-1% Success Rate & EF) (primary) — range: [0, 1]
  • Success rate: percentage of targets where the true binder is in the top 1% of candidates. Enhancement Factor (EF): ratio of true binders in the top 1% to the expected ratio if randomly selected.

Input / output format

Input: 3D atomic coordinates and connectivity for protein-ligand complexes (native or computationally generated decoy poses), ligand molecular graph/SMILES, and protein sequence/structure.

Output: Predicted binding affinity score, ranked list of ligand poses, or binary/continuous screening score for each candidate molecule.

Scoring recipe

R = pearsonr(predicted_affinities, experimental_affinities)
rho = spearmanr(predicted_affinities, experimental_affinities)

docking_success = sum(1 for c in complexes if native_pose_rank[c] == 1) / len(complexes)

top_k = max(1, int(0.01 * len(candidates)))
screening_success = sum(1 for t in targets if true_binder_rank[t] <= top_k) / len(targets)
EF = screening_success / 0.01

Common pitfalls

• High scoring power does not guarantee high docking or screening power, as models often overfit to true-binding complexes and fail on decoys.
• Scoring power alone is insufficient for evaluation; models must be tested on computationally generated decoys to assess generalization.
• Enhancement factor and success rate thresholds (e.g., top 1%) must be explicitly stated, as they vary across studies and significantly impact reported performance.

Evidence (verbatim from paper)

The scoring power measures a linear correlation of predicted binding affinities and experimental values, calculated by a Pearson's correlation coefficient R. The ranking power measures an ability of a model to correctly rank the binding affinities of true binders of the actual binding pose, calculated by a Spearman's rank-correlation coefficient ρ. These two metrics are designed to assess the model's ability upon the stable-and-precise binding structures. On the other hand, the docking power and the screening power deal with the unnatural structures which are generated computationally. The docking power measures an ability of a model to find out the native binding pose of a ligand among computer-generated decoys, quantified as a success rate within the top N candidates. The screening power measures the ability of a model to identify the specific binding ligand for a given target protein among a set of random molecules, quantified as a success rate and an enhancement factor (EF) within the top α percent of candidates.

Citation

@misc{moon2020pignet,
  title={PIGNet: A physics-informed deep learning model toward generalized drug-target interaction predictions},
  author={Moon et al. (2020)},
  year={2020},
  note={arXiv:2008.12249}
}

• arXiv: 2008.12249