Crossdocked Sbdd Eval

crossdocked-sbdd-eval

TacoGFN: Target-conditioned GFlowNet for Structure-based Drug Design — Shen et al. (2023) (arXiv:2310.03223, 2023)

What this evaluates

Evaluates a model's ability to generate novel, drug-like molecules with high binding affinity for unseen protein pockets in structure-based drug design. It probes the trade-offs between binding energy, molecular properties, and synthesis feasibility.

Datasets

• CrossDocked-100k — total 100000; splits: train (-1), test (100)

Metrics

• Validity — range: percent
  • Percentage of unique generated molecules that are free of reconstruction errors and disconnections, validated using RDKit.
• Vina Dock (primary) — range: other
  • Approximates the binding energy between a generated molecule and a protein pocket using AutoDock Vina; lower scores indicate higher binding affinity.
• High Affinity — range: percent
  • Percentage of generated molecules that achieve a higher binding affinity (lower Vina Dock score) than the reference/native molecule.
• QED — range: [0, 1]
  • Quantitative Estimate of Drug-likeness; estimates a molecule's suitability as an oral drug based on physicochemical properties.
• SA — range: [0, 1]
  • Synthetic Accessibility score estimating synthesis difficulty, normalized to [0, 1] using the formula (10 - SA)/9.
• Diversity — range: [0, 1]
  • Average pairwise Tanimoto distance calculated from molecular fingerprints of the generated set.
• Success Rate — range: percent
  • Percentage of generated molecules that simultaneously satisfy QED > 0.25, SA > 0.59, and Vina Dock < -8.18.
• Time — range: other
  • Average runtime in seconds required to generate 100 unique and valid molecules for a single protein pocket.

Input / output format

Input: 3D protein pocket structure/coordinates

Output: 100 generated molecule structures (ligands) per protein pocket

Scoring recipe

def compute_metrics(predictions, pocket, ref_dock):
    valid_mols = [m for m in predictions if rdkit.is_valid(m)]
    validity = len(valid_mols) / len(predictions)
    vina_scores = [dock_vina(m, pocket) for m in valid_mols]
    high_aff = sum(1 for s in vina_scores if s < ref_dock) / len(valid_mols)
    qed_vals = [calc_qed(m) for m in valid_mols]
    sa_vals = [(10 - calc_sa(m)) / 9 for m in valid_mols]
    diversity = avg_tanimoto_distance(valid_mols)
    success = sum(1 for m in valid_mols if calc_qed(m) > 0.25 and calc_sa(m) > 0.59 and dock_vina(m, pocket) < -8.18) / len(valid_mols)
    return validity, vina_scores, high_aff, qed_vals, sa_vals, diversity, success

Common pitfalls

• Heavy molecules often achieve better Vina Dock scores but violate drug-likeness (QED) and Rule of 5 constraints, creating a trade-off that skews naive docking optimization.
• Success rate thresholds (QED > 0.25, SA > 0.59, Vina Dock < -8.18) are strict and specific to this benchmark; relaxing them significantly inflates reported performance.
• Diversity and reward quality trade off based on sampling temperature β; higher β yields better affinity but lower diversity, requiring careful temperature selection for fair comparison.

Evidence (verbatim from paper)

In all evaluations, each structure-based generative model is tasked to produce 100 molecules (ligands) for each of the 100 unseen protein pockets from the CrossDock-100k test set. Evaluation metrics. We adopt the following commonly used metrics from Guan et al. (2023a) and Reidenbach (2024): (1) Validity is the percentage of unique generated molecules free of reconstruction errors and disconnections as determined by RDKit. (2) Vina Dock approximates the binding energy between a generated molecule and a protein pocket, where a lower docking score indicates a higher binding affinity. (7) Success Rate is the percentage of molecules which pass the same criteria (QED > 0.25, SA > 0.59, Vina Dock < -8.18) as in Long et al. (2022); Guan et al. (2023b); Zhou et al. (2024); Reidenbach (2024).

Citation

@misc{shen2023tacogfn,
  title={TacoGFN: Target-conditioned GFlowNet for Structure-based Drug Design},
  author={Shen et al. (2023)},
  year={2023},
  note={arXiv:2310.03223}
}

• arXiv: 2310.03223