# Smiles Tokenization And Canonical Representation

> Use when when preparing SMILES strings as training targets for a sequence-to-sequence decoder that reconstructs molecular structures from embeddings.

- Skill: `holobiomicslab/smiles-tokenization-and-canonical-representation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/smiles-tokenization-and-canonical-representation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/smiles-tokenization-and-canonical-representation/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/smiles-tokenization-and-canonical-representation

---


# SMILES tokenization and canonical representation

## Summary

Convert molecular structures into canonical SMILES strings and tokenize them into variable-length sequences for use as training targets in sequence-to-sequence models. This skill ensures consistent, unambiguous molecular representation suitable for neural decoder training.

## When to use

When preparing SMILES strings as training targets for a sequence-to-sequence decoder that reconstructs molecular structures from embeddings. Apply this skill when you need to standardize molecular representations before feeding them into cross-entropy loss calculations or when comparing reconstructed molecules against reference structures.

## When NOT to use

- Input molecules are already in a standardized canonical form and exact-match accuracy is not required for your application.
- Your decoder is designed to output molecular representations other than SMILES (e.g., SELFIES, molecular graphs, or InChI strings).
- You are evaluating pre-trained decoders on non-molecular sequence tasks where SMILES canonicalization is not applicable.

## Inputs

- SMILES strings (variable-length character sequences representing molecular structures)
- Molecular structure objects (e.g., RDKit Mol objects)
- Decoder embeddings (fixed-size vector representations from encoder)

## Outputs

- Canonical SMILES strings (standardized, unambiguous molecular representations)
- Tokenized SMILES sequences (variable-length integer or token ID sequences)
- SMILES token vocabulary (mapping of unique tokens to integer indices)
- Reconstructed SMILES strings (predicted from decoder output)

## How to apply

Use RDKit to canonicalize SMILES strings, ensuring each molecule has a unique, standardized representation that eliminates isomeric ambiguity. Tokenize the canonical SMILES into individual token sequences (atoms, bonds, brackets, etc.) to create variable-length token sequences aligned with the decoder's output vocabulary. Apply this preprocessing to both training and held-out test SMILES datasets. During training, use teacher forcing with these tokenized sequences as targets for cross-entropy loss computation on SMILES token prediction. During evaluation, reconstruct molecules from decoder predictions by joining tokens back into SMILES strings, then parse them with RDKit to compute exact-match accuracy and Tanimoto similarity against reference molecules.

## Related tools

- **RDKit** (Parse, canonicalize, and validate SMILES strings; reconstruct molecules from SMILES predictions to compute Tanimoto similarity metrics) — https://www.rdkit.org/
- **PyTorch** (Implement tokenization indices in embedding lookup layers and compute cross-entropy loss on SMILES token predictions during decoder training)

## Examples

```
# Canonicalize SMILES and prepare tokenized sequences for decoder training
from rdkit import Chem
import torch

# Load SMILES strings and canonicalize
smiles_list = ['CC(C)Cc1ccc(cc1)C(C)C(O)=O', 'c1ccc(cc1)C(C)C(O)=O']
canonical_smiles = [Chem.MolToSmiles(Chem.MolFromSmiles(s)) for s in smiles_list]

# Tokenize and create vocabulary
vocab = {'C': 0, '(': 1, ')': 2, '=': 3, 'O': 4, 'c': 5, '1': 6}  # example
tokenized = [[vocab[t] for t in s] for s in canonical_smiles]

# Use in training with teacher forcing
for tokens in tokenized:
    loss = criterion(decoder_output, torch.tensor(tokens))
```

## Evaluation signals

- Canonical SMILES strings are consistent across multiple invocations on the same molecule (e.g., different input SMILES isomers produce identical canonical output).
- All tokens in the tokenized sequences map to entries in the vocabulary; no out-of-vocabulary tokens appear in training or validation sets.
- Exact-match accuracy on held-out test set: reconstructed SMILES strings, when canonicalized, match reference SMILES exactly.
- Tanimoto similarity of reconstructed vs. reference molecules (computed via RDKit fingerprints) is ≥ 0.7 for true-positive predictions, indicating structural coherence.
- Cross-entropy loss converges during decoder training when teacher forcing uses these tokenized SMILES as targets.

## Limitations

- RDKit canonicalization may fail or produce unexpected results for uncommon or invalid SMILES strings; pre-filtering for valid molecules is recommended.
- Tokenization is vocabulary-dependent; if the training set contains rare or novel chemical substructures, the fixed vocabulary may not capture all valid tokens from test molecules.
- Exact-match accuracy is strict and does not account for multiple valid SMILES representations of the same molecule; Tanimoto similarity is a more lenient metric but depends on fingerprint choice.
- No changelog was found in the Spec2Mol repository, so historical changes to SMILES processing or tokenization schemes are not documented.

## Evidence

- [methods] Canonicalization via RDKit: "Prepare SMILES tokenizer and target SMILES string dataset, converting molecules to canonical SMILES format using RDKit."
- [methods] Tokenization for sequence models: "Define the decoder architecture in PyTorch as a sequence-to-sequence model with attention, mapping fixed-size embeddings to variable-length SMILES token sequences."
- [methods] Cross-entropy loss on tokens: "Train the decoder end-to-end using cross-entropy loss on SMILES token prediction, with teacher forcing during training."
- [methods] Reconstruction and evaluation: "Evaluate decoder on held-out test embeddings by measuring exact-match accuracy and Tanimoto similarity of reconstructed vs. reference molecules (decoded via RDKit)."
- [readme] RDKit chemical data processing: "Processing of the chemical data is based on the RDKit software."