Covid Drug Design Eval

covid-drug-design-eval

Benchmarking Deep Graph Generative Models for Optimizing New Drug Molecules for COVID-19 — Ward et al. (2021) (arXiv:2102.04977, 2021)

What this evaluates

This benchmark evaluates deep graph generative models (JT-VAE and DQN) for their ability to design novel molecular structures optimized for high predicted potency against the SARS-CoV-2 3CL-protease, while balancing drug-likeness, lipophilicity, and synthesizability. It also assesses structural novelty relative to known antivirals and predicted binding affinity using in silico classifiers.

Datasets

• ChEMBL/BindingDB/ToxCat pharmacology dataset — total ?; splits: train (-1); repo https://github.com/exalearn/covid-drug-design

Metrics

• pIC50 (primary) — range: other
  • Predicted negative logarithm of the half-maximal inhibitory concentration against SARS-CoV-2 3CL-protease, estimated via a Message Passing Neural Network (MPNN) surrogate model trained on experimental pharmacology data.
• QED — range: [0, 1]
  • Quantitative Estimate of Drug-likeness, a composite score balancing molecular properties like logP, molecular weight, and hydrogen bond counts.
• logP — range: other
  • Octanol-water partition coefficient measuring lipophilicity.
• SA Score — range: other
  • Synthetic Accessibility Score estimating how easy a molecule is to synthesize based on molecular complexity and synthetic feasibility.
• DBTA Probability — range: [0, 1]
  • Predicted probability of drug-target binding interaction, output by a Convolutional Neural Network classifier using extended connectivity fingerprints and protease sequence encoding.
• Tanimoto Similarity — range: [0, 1]
  • Structural similarity coefficient between generated molecules and the reference drug Indinavir, calculated using molecular fingerprints.

Input / output format

Input: Molecular graphs (atomic structures/bonds) for generation; for evaluation, generated molecules are encoded as molecular fingerprints or graphs and fed into surrogate models (MPNN for pIC50, CNN for DBTA) or standard property calculators.

Output: Generated molecular structures (represented as graphs/SMILES) and their corresponding property scores (pIC50, QED, logP, SA Score, DBTA probability, Tanimoto similarity).

Scoring recipe

def evaluate_molecules(generated_mols, mpnn_model, dbta_model, indinavir_fp):
    results = []
    for mol in generated_mols:
        scores = {}
        scores['pIC50'] = mpnn_model.predict(mol)
        scores['QED'] = qed_calculator(mol)
        scores['logP'] = logp_calculator(mol)
        scores['SA'] = sa_calculator(mol)
        scores['DBTA'] = dbta_model.predict(mol, target_protein)
        scores['Tanimoto'] = tanimoto_similarity(mol, indinavir_fp)
        results.append(scores)
    # Report top-3 highest scores per metric per model/scoring function
    return top_n(results, n=3)

Common pitfalls

• pIC50 and DBTA are in silico predictions from surrogate models, not experimental biological assays, so they may not reflect true in vivo potency or binding.
• The evaluation relies on default hyperparameters from external toolkits (DeepPurpose for DBTA) without ablation, which may limit reproducibility or generalizability.
• QED, logP, and SA scores are standard but not explicitly defined with formulas in the text; readers must assume standard RDKit implementations.

Evidence (verbatim from paper)

We trained a DBTA binary classification model using extended connectivity fingerprint*(Rogers and Hahn, 2010)* encoding for the drug molecule and the target protease sequence encoding using a Convolution Neural Network (CNN) as implemented in the DeepPurpose toolkit*(Huang et al., 2020)*. The default hyperparameters provided in the DeepPurpose toolkit were found to be sufficient. The DBTA model classified four of the top 11 molecules (including the top two in Figure 3) with probability > 0.5 to have interaction with SARS-CoV2-3CL protease.

Citation

@misc{ward2021benchmarking,
  title={Benchmarking Deep Graph Generative Models for Optimizing New Drug Molecules for COVID-19},
  author={Ward et al. (2021)},
  year={2021},
  note={arXiv:2102.04977}
}

• arXiv: 2102.04977