Target Aware Molecular Generation Eval

target-aware-molecular-generation-eval

Target-aware Molecular Graph Generation — Anonymous et al. (2022) (arXiv:2202.04829, 2022)

What this evaluates

Evaluates the ability of conditional generative models to produce chemically valid and structurally similar molecular graphs conditioned on specific target protein sequences. It probes both the generative quality (validity, uniqueness, novelty) and the chemical fidelity (similarity to known drugs) of the generated molecules.

Datasets

• Combined Drug-Target Dataset (BIOSNAP, BindingDB, DAVIS, DrugBank) — total 24669; splits: train (-1), val (-1), test (-1)

Metrics

• validity — range: percent
  • Percentage of generated molecules that are chemically valid according to RDKit.
• uniqueness — range: percent
  • Percentage of unique valid molecules among all generated molecules.
• novelty — range: percent
  • Percentage of generated valid molecules that do not appear in the training dataset.
• Tanimoto similarity (primary) — range: percent
  • Tanimoto coefficient calculated based on hashed binary molecular fingerprints.
• Fraggle similarity — range: percent
  • Fragment-level similarity score between generated and reference molecules.
• MACCS similarity — range: percent
  • Similarity score computed using 166-bit 2D MACCS fingerprints.

Input / output format

Input: Target protein sequence (string) used as conditioning input for molecule generation.

Output: Generated molecular graph representation (e.g., SMILES string or atom/bond graph) corresponding to the target.

Scoring recipe

def compute_metrics(generated_smiles, train_smiles_set):
    valid = [s for s in generated_smiles if rdkit_mol_from_smiles(s) is not None]
    validity = len(valid) / len(generated_smiles)
    uniqueness = len(set(valid)) / len(generated_smiles)
    novelty = len([s for s in valid if s not in train_smiles_set]) / len(valid)
    tanimoto = mean(tanimoto_sim(gen_mol, nearest_train_mol) for gen_mol in valid)
    fraggle = mean(fraggle_sim(gen_mol, nearest_train_mol) for gen_mol in valid)
    maccs = mean(maccs_sim(gen_mol, nearest_train_mol) for gen_mol in valid)
    return validity, uniqueness, novelty, tanimoto, fraggle, maccs

Common pitfalls

• Sequence-to-sequence baselines often achieve low validity/uniqueness due to memorization rather than true generation.
• MACCS similarity can be artificially inflated for sequence-partition models, making structure-centric metrics like Tanimoto and Fraggle more reliable for assessing chemical fidelity.
• The zero-overlap protein split requires strict dataloader implementation to prevent data leakage between train/val/test sets.

Evidence (verbatim from paper)

To comprehensively evaluate the conditional generative models in terms of target-aware molecular generation, we design metrics from two perspectives: (1) Generative metrics. Following the common molecular generation settings, we apply metrics including: validity which is the percentage of chemically valid molecules in all the generated molecules, uniqueness which is the percentage of unique valid molecules in all the generated molecules, novelty which is the percentage of generated valid molecules which are not in the training dataset. (2) Chemical metrics. We evaluate the similarities between the generated drugs and the nearest drugs in the training set including: Tanimoto similarity which is calculated based on hashed binary features, Fraggle similarity which focus on the fragment-level similarity, MACCS similarity which employs 166-bit 2D structure fingerprints.

Citation

@misc{anonymous2022targetaware,
  title={Target-aware Molecular Graph Generation},
  author={Anonymous et al. (2022)},
  year={2022},
  note={arXiv:2202.04829}
}

• arXiv: 2202.04829