Molecule Design Eval

molecule-design-eval

Multi-Objective Molecule Generation using Interpretable Substructures — Wengong Jin, Regina Barzilay, Tommi Jaakkola (2020) (arXiv:2002.03244, 2020)

What this evaluates

Evaluates the ability of generative models to design molecules that satisfy multiple property constraints (e.g., biological activity, drug-likeness, synthetic accessibility) while maintaining chemical diversity and novelty. It also assesses the faithfulness of extracted substructure rationales in explaining target properties.

Datasets

• GSK3β — total 52665; splits: train (-1), val (-1), test (-1)
• JNK3 — total 50740; splits: train (-1), val (-1), test (-1)
• Toxicity — total 125000; splits: train (-1), test (-1)

Metrics

• Success (primary) — range: [0, 1]
  • The fraction of sampled molecules predicted to be positive by the property predictor (threshold > 0.5), indicating how well the model satisfies the given property constraints.
• Diversity — range: [0, 1]
  • 1 - (2 / (n(n-1))) * sum(Tanimoto(X, Y)) over all unique pairs of generated molecules, where Tanimoto distance is computed over Morgan fingerprints.
• Novelty — range: [0, 1]
  • The fraction of generated molecules whose nearest neighbor in the training positive set has a Tanimoto similarity lower than 0.4.

Input / output format

Input: Rationale substructure graphs and target property constraints (e.g., GSK3β inhibition, QED ≥ 0.6, SA ≤ 4.0).

Output: Generated molecular graphs (or SMILES strings) intended to satisfy the specified constraints.

Scoring recipe

def compute_metrics(generated_mols, train_positives, predictor):
    n = len(generated_mols)
    success = sum(1 for m in generated_mols if predictor.predict(m) > 0.5) / n
    diversity = 1.0 - (2.0 / (n * (n - 1))) * sum(tanimoto(m1, m2) for m1, m2 in combinations(generated_mols))
    novelty = sum(1 for m in generated_mols if min(tanimoto(m, tp) for tp in train_positives) < 0.4) / n
    return {'success': success, 'diversity': diversity, 'novelty': novelty}

Common pitfalls

• Success rate is evaluated exclusively using in silico property predictors (random forests on Morgan fingerprints) rather than wet-lab assays, which may overestimate performance due to predictor flaws or distribution shift.
• Novelty relies on an arbitrary similarity threshold (0.4) against the training set, which may not align with standard chemical novelty benchmarks or could exclude valid new scaffolds.
• Diversity metric is sensitive to Morgan fingerprint parameters (radius, bit length) that are not always standardized across different baseline implementations.

Evidence (verbatim from paper)

Our evaluation effort measures various aspects of molecule design. For each method, we generate $n=5000$ molecules and compute the following metrics: • Success: The fraction of sampled molecules predicted to be positive (i.e., satisfying all property constraints). A good model should have a high success rate. Following previous work, we only consider the success rate under property predictors, as it is hard to obtain real property measurements for GSK3$\beta$ and JNK3. • Diversity: It is also important for a model generate diverse range of positive molecules. To this end, we measure the diversity of generated positive compounds by computing their pairwise molecular distance $\mathrm{sim}(X,Y)$, which is defined as the Tanimoto distance over Morgan fingerprints of two molecules. • Novelty: Crucially, a good model should discover novel positive compounds. In this regard, for each generated positive compound $\mathcal{G}$, we find its nearest neighbor $\mathcal{G}_{\mathrm{SNN}}$ from positive molecules in the training set. We define the novelty as the fraction of molecules with nearest neighbor similarity lower than 0.4

Citation

@misc{jin2020multiobjective,
  title={Multi-Objective Molecule Generation using Interpretable Substructures},
  author={Wengong Jin, Regina Barzilay, Tommi Jaakkola (2020)},
  year={2020},
  note={arXiv:2002.03244}
}

• arXiv: 2002.03244