Generative Molecular Design
De novo design of novel molecules with desired properties using generative models — the core ML capability for lead generation and scaffold hopping in drug discovery.
When to Use This Skill
- Generate molecules with target properties (QED, LogP, SA, docking score)
- Explore chemical space around a hit/lead (analogue generation, scaffold hopping)
- Design molecules conditioned on a protein pocket (SBDD)
- Optimize multi-property objectives (Pareto front: potency + selectivity + ADMET)
- Benchmark or compare generative models (MOSES / GuacaMol suites)
- Build a RL-based focused library generator (REINVENT 4)
- Design linkers or grow fragments (fragment-based generative design)
Generation Paradigms
| Paradigm |
Method |
Strength |
Weakness |
| Language model |
SMILES/SELFIES GPT, LSTM |
Fast, scalable, fine-tunable |
SMILES can be invalid; needs SELFIES |
| VAE |
JT-VAE, MolVAE |
Smooth latent space, BO-ready |
Mode collapse; slow tree encode |
| GNN flow/GAN |
GraphAF, GCPN, JunctionGAN |
Graph-native; no linearity |
Training instability |
| RL optimization |
REINVENT 4, REINFORCE |
Property-guided; no new arch needed |
Reward hacking; mode collapse |
| 3D diffusion |
DiffSBDD, TargetDiff |
Pocket-conditioned; 3D geometry |
Slow, needs structure |
| Fragment-based |
DeLinker, DiffLinker |
Fragment growing, FBDD |
Limited to provided fragments |
Evaluation Metrics (Know These)
| Metric |
What it measures |
Target |
| Validity |
% chemically valid |
~100% (SELFIES) / 85-99% (SMILES LM) |
| Uniqueness |
% unique in generated set |
>99% |
| Novelty |
% not in training set |
>99% |
| FCD |
Fréchet ChemNet Distance (distribution) |
Lower = closer to drug-like distribution |
| KL divergence |
Property distributions vs. reference |
Lower |
| Scaffold diversity |
# unique Murcko scaffolds / N |
Higher |
| IntDiv |
Internal diversity (mean pairwise 1-Tc) |
> 0.85 |
| SNN |
Similarity to nearest neighbor in training |
< 0.6 (novel) |
Quick Start — SELFIES + GPT sampling
import selfies as sf
from rdkit import Chem
# Encode/decode SELFIES (guaranteed valid)
smiles = "CC(=O)Oc1ccccc1C(=O)O" # aspirin
selfies_str = sf.encoder(smiles)
decoded_smiles = sf.decoder(selfies_str)
mol = Chem.MolFromSmiles(decoded_smiles) # always valid
# Get SELFIES alphabet for tokenization
alphabet = sf.get_semantic_robust_alphabet()
# Decode a random generated SELFIES token sequence (always valid):
generated_tokens = ["[C]", "[Branch1]", "[C]", "[=O]", "[N]", "[H]"]
generated_smiles = sf.decoder("".join(generated_tokens))
Quick Start — REINVENT 4 scoring component
# Install: pip install reinvent
# REINVENT 4 uses TOML config for staged learning
import toml
config = {
"run_type": "reinforcement_learning",
"device": "cuda",
"tb_logdir": "tb_logs",
"json_out_config": "run_config.json",
"parameters": {
"use_checkpoint": False,
"prior_file": "path/to/prior.prior",
"agent_file": "path/to/prior.prior",
"batch_size": 128,
"n_steps": 1000,
},
"scoring": {
"type": "custom_product",
"parallel": False,
"components": [
{"type": "qed", "name": "QED", "weight": 1.0},
{"type": "sa_score", "name": "SA", "weight": 1.0,
"transform": {"type": "reverse_sigmoid", "low": 1.0, "high": 6.0, "k": 0.5}},
],
"diversity_filter": {
"type": "IdenticalMurckoScaffold",
"minscore": 0.4,
"bucket_size": 25,
}
}
}
with open("rl_config.toml", "w") as f:
toml.dump(config, f)
# Run: reinvent -l rl_run.log rl_config.toml
Router — What to Read
| Task |
Reference |
| Theory: molecular space, SMILES/SELFIES/graphs, metrics, MOSES/GuacaMol benchmarks |
references/generation-theory.md |
| SELFIES grammar, SMILES LM (GPT/LSTM), HuggingFace fine-tuning, sampling strategies |
references/selfies-lm.md |
| REINVENT 4: RL optimization, multi-component scoring, diversity filters, oracles |
references/rl-reinvent.md |
| JT-VAE: tree decomposition, latent BO; TorchDrug graph generative models overview |
references/vae-jtvae.md |
| Structure-based 3D generation: DiffSBDD, TargetDiff, Pocket2Mol, linker design |
references/sbdd-diffusion.md |
Software Stack
| Package |
Install |
Role |
selfies |
pip install selfies |
Always-valid molecular grammar |
reinvent |
pip install reinvent |
RL de novo design (AZ REINVENT 4) |
guacamol |
pip install guacamol |
Benchmark suite (17 goal-directed + distributional) |
moses |
pip install molsets |
MOSES benchmark (6 metrics) |
transformers |
pip install transformers |
GPT/LSTM LMs (HuggingFace) |
torchdrug |
pip install torchdrug |
GCPN, GraphAF, JT-VAE (graph-native) |
DiffSBDD |
GitHub: arneschneuing/DiffSBDD |
3D pocket-conditioned diffusion |
DiffLinker |
GitHub: igashov/DiffLinker |
Linker design in 3D |
Key Pitfalls
- SMILES LMs can generate 10-50% invalid → use SELFIES; or add validity filter post-hoc
- Reward hacking in RL: model learns degenerate structures that maximize score — add diversity filter + SA penalty
- FCD is not computed from structure: requires ChemNet embeddings (guacamol includes this)
- Novelty ≠ synthesizability: always check SA score ≤ 4, run retrosynthesis (ASKCOS/AiZynthFinder)
- 3D diffusion needs pocket quality: must use properly prepared protein (see
homology-modeling → structure-prep)
- Mode collapse in VAE: monitor KL weight β; schedule β-VAE warmup
Related Skills
torchdrug — GCPN, GraphAF, GraphDF, JT-VAE implementation
rdkit — validity checks, property scoring oracles (QED, SA, fingerprints)
docking — docking oracle for RL scoring (Vina/Gnina scoring function)
pharmacophore — pharmacophore constraints for conditional generation
homology-modeling → structure-prep — pocket preparation for SBDD
scientific-skills:zinc-database — training/reference sets (ZINC20)
mmpa (upcoming) — matched molecular pair analysis on generated series
1---2name: generative-design3description: Use when designing or evaluating generative models for de novo drug/molecule design. Covers molecular generation theory and evaluation (MOSES/GuacaMol), SELFIES + language models, RL-based optimization with REINVENT 4, JT-VAE and graph-based generation, and structure-based 3D generation (DiffSBDD, Pocket2Mol, DiffLinker).4---56# Generative Molecular Design78De novo design of novel molecules with desired properties using generative models — the core ML capability for lead generation and scaffold hopping in drug discovery.910## When to Use This Skill1112- Generate molecules with target properties (QED, LogP, SA, docking score)13- Explore chemical space around a hit/lead (analogue generation, scaffold hopping)14- Design molecules conditioned on a protein pocket (SBDD)15- Optimize multi-property objectives (Pareto front: potency + selectivity + ADMET)16- Benchmark or compare generative models (MOSES / GuacaMol suites)17- Build a RL-based focused library generator (REINVENT 4)18- Design linkers or grow fragments (fragment-based generative design)1920## Generation Paradigms2122| Paradigm | Method | Strength | Weakness |23|----------|--------|----------|----------|24| Language model | SMILES/SELFIES GPT, LSTM | Fast, scalable, fine-tunable | SMILES can be invalid; needs SELFIES |25| VAE | JT-VAE, MolVAE | Smooth latent space, BO-ready | Mode collapse; slow tree encode |26| GNN flow/GAN | GraphAF, GCPN, JunctionGAN | Graph-native; no linearity | Training instability |27| RL optimization | REINVENT 4, REINFORCE | Property-guided; no new arch needed | Reward hacking; mode collapse |28| 3D diffusion | DiffSBDD, TargetDiff | Pocket-conditioned; 3D geometry | Slow, needs structure |29| Fragment-based | DeLinker, DiffLinker | Fragment growing, FBDD | Limited to provided fragments |3031## Evaluation Metrics (Know These)3233| Metric | What it measures | Target |34|--------|-----------------|--------|35| Validity | % chemically valid | ~100% (SELFIES) / 85-99% (SMILES LM) |36| Uniqueness | % unique in generated set | >99% |37| Novelty | % not in training set | >99% |38| **FCD** | Fréchet ChemNet Distance (distribution) | Lower = closer to drug-like distribution |39| KL divergence | Property distributions vs. reference | Lower |40| Scaffold diversity | # unique Murcko scaffolds / N | Higher |41| IntDiv | Internal diversity (mean pairwise 1-Tc) | > 0.85 |42| SNN | Similarity to nearest neighbor in training | < 0.6 (novel) |4344## Quick Start — SELFIES + GPT sampling4546```python47import selfies as sf48from rdkit import Chem4950# Encode/decode SELFIES (guaranteed valid)51smiles = "CC(=O)Oc1ccccc1C(=O)O" # aspirin52selfies_str = sf.encoder(smiles)53decoded_smiles = sf.decoder(selfies_str)54mol = Chem.MolFromSmiles(decoded_smiles) # always valid5556# Get SELFIES alphabet for tokenization57alphabet = sf.get_semantic_robust_alphabet()5859# Decode a random generated SELFIES token sequence (always valid):60generated_tokens = ["[C]", "[Branch1]", "[C]", "[=O]", "[N]", "[H]"]61generated_smiles = sf.decoder("".join(generated_tokens))62```6364## Quick Start — REINVENT 4 scoring component6566```python67# Install: pip install reinvent68# REINVENT 4 uses TOML config for staged learning6970import toml7172config = {73 "run_type": "reinforcement_learning",74 "device": "cuda",75 "tb_logdir": "tb_logs",76 "json_out_config": "run_config.json",77 "parameters": {78 "use_checkpoint": False,79 "prior_file": "path/to/prior.prior",80 "agent_file": "path/to/prior.prior",81 "batch_size": 128,82 "n_steps": 1000,83 },84 "scoring": {85 "type": "custom_product",86 "parallel": False,87 "components": [88 {"type": "qed", "name": "QED", "weight": 1.0},89 {"type": "sa_score", "name": "SA", "weight": 1.0,90 "transform": {"type": "reverse_sigmoid", "low": 1.0, "high": 6.0, "k": 0.5}},91 ],92 "diversity_filter": {93 "type": "IdenticalMurckoScaffold",94 "minscore": 0.4,95 "bucket_size": 25,96 }97 }98}99with open("rl_config.toml", "w") as f:100 toml.dump(config, f)101# Run: reinvent -l rl_run.log rl_config.toml102```103104## Router — What to Read105106| Task | Reference |107|------|-----------|108| Theory: molecular space, SMILES/SELFIES/graphs, metrics, MOSES/GuacaMol benchmarks | `references/generation-theory.md` |109| SELFIES grammar, SMILES LM (GPT/LSTM), HuggingFace fine-tuning, sampling strategies | `references/selfies-lm.md` |110| REINVENT 4: RL optimization, multi-component scoring, diversity filters, oracles | `references/rl-reinvent.md` |111| JT-VAE: tree decomposition, latent BO; TorchDrug graph generative models overview | `references/vae-jtvae.md` |112| Structure-based 3D generation: DiffSBDD, TargetDiff, Pocket2Mol, linker design | `references/sbdd-diffusion.md` |113114## Software Stack115116| Package | Install | Role |117|---------|---------|------|118| `selfies` | `pip install selfies` | Always-valid molecular grammar |119| `reinvent` | `pip install reinvent` | RL de novo design (AZ REINVENT 4) |120| `guacamol` | `pip install guacamol` | Benchmark suite (17 goal-directed + distributional) |121| `moses` | `pip install molsets` | MOSES benchmark (6 metrics) |122| `transformers` | `pip install transformers` | GPT/LSTM LMs (HuggingFace) |123| `torchdrug` | `pip install torchdrug` | GCPN, GraphAF, JT-VAE (graph-native) |124| `DiffSBDD` | GitHub: arneschneuing/DiffSBDD | 3D pocket-conditioned diffusion |125| `DiffLinker` | GitHub: igashov/DiffLinker | Linker design in 3D |126127## Key Pitfalls128129- **SMILES LMs can generate 10-50% invalid** → use SELFIES; or add validity filter post-hoc130- **Reward hacking in RL**: model learns degenerate structures that maximize score — add diversity filter + SA penalty131- **FCD is not computed from structure**: requires ChemNet embeddings (guacamol includes this)132- **Novelty ≠ synthesizability**: always check SA score ≤ 4, run retrosynthesis (ASKCOS/AiZynthFinder)133- **3D diffusion needs pocket quality**: must use properly prepared protein (see `homology-modeling` → structure-prep)134- **Mode collapse in VAE**: monitor KL weight β; schedule β-VAE warmup135136## Related Skills137138- `torchdrug` — GCPN, GraphAF, GraphDF, JT-VAE implementation139- `rdkit` — validity checks, property scoring oracles (QED, SA, fingerprints)140- `docking` — docking oracle for RL scoring (Vina/Gnina scoring function)141- `pharmacophore` — pharmacophore constraints for conditional generation142- `homology-modeling` → `structure-prep` — pocket preparation for SBDD143- `scientific-skills:zinc-database` — training/reference sets (ZINC20)144- `mmpa` (upcoming) — matched molecular pair analysis on generated series