Sequence To Sequence Architecture Design

sequence-to-sequence-architecture-design

Summary

Design and implement a sequence-to-sequence (seq2seq) model with attention that maps fixed-size embeddings to variable-length token sequences, such as reconstructing SMILES strings from molecular embeddings. This skill bridges continuous latent representations to discrete symbolic outputs using an encoder-decoder paradigm with teacher forcing during training.

When to use

When you have encoder-produced fixed-size embeddings and need to generate variable-length discrete sequences (e.g., SMILES tokens, protein sequences, chemical formulas) as outputs. Specifically applicable when the input is a continuous embedding and the output is a sequence of categorical tokens with no predetermined length.

When NOT to use

• Input is already a sequence (use sequence-to-sequence without an embedding bottleneck instead).
• Output is fixed-length or continuous-valued (use a fully connected regression head or classifier instead).
• Target sequences are very short (single tokens) or deterministic transformations (use a simple lookup table or rule-based decoder).

Inputs

• encoder-produced embeddings (fixed-size continuous vectors)
• SMILES token vocabulary or domain-specific tokenizer
• target sequence dataset (e.g., canonical SMILES strings, protein sequences)

Outputs

• trained decoder model weights
• decoded variable-length sequences (e.g., SMILES strings)
• validation metrics (exact-match accuracy, Tanimoto similarity, cross-entropy loss)

How to apply

Define a sequence-to-sequence model in PyTorch with an embedding layer that accepts the fixed-size encoder output, followed by an attention-equipped recurrent decoder (e.g., LSTM or GRU) that predicts one token at a time. Prepare a tokenizer for your target domain (e.g., SMILES tokenizer via RDKit) and convert all target sequences to canonical representations. Train the decoder end-to-end using cross-entropy loss on token prediction with teacher forcing (feeding ground-truth tokens during training rather than model predictions). Evaluate using exact-match accuracy on held-out embeddings and domain-specific similarity metrics (e.g., Tanimoto similarity between decoded and reference molecules via RDKit). Log validation metrics and save trained decoder weights separately from the encoder for modularity.

Related tools

• PyTorch (framework for implementing sequence-to-sequence model architecture with attention, cross-entropy loss computation, and backpropagation training)
• RDKit (tokenization of SMILES strings, canonicalization of molecular structures, and computation of Tanimoto similarity for evaluation) — https://www.rdkit.org/

Examples

decoder = Seq2SeqDecoder(embedding_dim=256, vocab_size=120, hidden_dim=512, num_layers=2, attention=True); criterion = nn.CrossEntropyLoss(); optimizer = torch.optim.Adam(decoder.parameters(), lr=0.001); for epoch in range(num_epochs): loss = train_epoch(decoder, train_loader, criterion, optimizer, teacher_forcing_ratio=1.0); val_acc, val_tanimoto = evaluate(decoder, val_loader); print(f'Epoch {epoch}: train_loss={loss:.4f}, val_acc={val_acc:.4f}, val_tanimoto={val_tanimoto:.4f}')

Evaluation signals

• Exact-match accuracy on held-out test embeddings (reconstructed SMILES == reference SMILES without RDKit canonicalization errors).
• Tanimoto similarity between decoded and reference molecules (via RDKit fingerprints; expect high similarity if decoding is faithful).
• Cross-entropy loss converging and validation loss decreasing over training epochs.
• All generated sequences are valid tokens according to the target tokenizer vocabulary.
• Decoder output sequences vary in length as expected (not all identical lengths) and respect chemical feasibility constraints when decoded to molecules.

Limitations

• Teacher forcing during training can cause exposure bias: at inference, the decoder conditions on its own (potentially erroneous) predictions rather than ground truth, degrading sequence quality over long horizons.
• Attention mechanism assumes the embedding is sufficiently informative; if the encoder embedding loses critical structural information, the decoder cannot recover it.
• Exact-match accuracy is brittle for SMILES (same molecule can have multiple valid SMILES representations); Tanimoto similarity or other fuzzy metrics are more robust but slower to compute.
• Training requires paired encoder embeddings and target sequences; unpaired or weakly labeled data cannot be used directly.

Evidence

• [other] Define the decoder architecture in PyTorch as a sequence-to-sequence model with attention, mapping fixed-size embeddings to variable-length SMILES token sequences.: "Define the decoder architecture in PyTorch as a sequence-to-sequence model with attention, mapping fixed-size embeddings to variable-length SMILES token sequences."
• [other] Train the decoder end-to-end using cross-entropy loss on SMILES token prediction, with teacher forcing during training.: "Train the decoder end-to-end using cross-entropy loss on SMILES token prediction, with teacher forcing during training."
• [other] Evaluate decoder on held-out test embeddings by measuring exact-match accuracy and Tanimoto similarity of reconstructed vs. reference molecules (decoded via RDKit).: "Evaluate decoder on held-out test embeddings by measuring exact-match accuracy and Tanimoto similarity of reconstructed vs. reference molecules (decoded via RDKit)."
• [readme] The decoder reconstructs the molecular structure, in a SMILES format, given the embedding that the encoder generates.: "The decoder reconstructs the molecular structure, in a SMILES format, given the embedding that the encoder generates."
• [readme] The implementation of the Spec2Mol architecture is based on the Pytorch library. Processing of the chemical data is based on the RDKit software.: "The implementation of the Spec2Mol architecture is based on the Pytorch library. Processing of the chemical data is based on the RDKit software."