PyTorch Graph Serialization for GNN Input
Summary
Serialize molecular graph representations into PyTorch-compatible formats (pickle or HDF5) to enable efficient consumption by Graph Neural Network training pipelines. This skill bridges the gap between RDKit molecular graph construction and PyTorch DataLoader ingestion for retention time prediction models.
When to use
You have constructed molecular graphs with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type, aromaticity) from SMILES or MOL files using RDKit, and need to feed them into a PyTorch-based GNN training pipeline without memory overhead or I/O bottlenecks during model training and transfer learning phases.
When NOT to use
- Input is already in PyTorch Tensor or graph object format (Data or HeteroData from PyG)
- Molecular graphs are extremely large (>10 million atoms per structure) and serialization overhead exceeds memory savings
- Training pipeline expects streaming/online data augmentation rather than pre-serialized static batches
Inputs
- Molecular graphs with node-edge representations (constructed via RDKit)
- Atom feature arrays (atomic number, degree, formal charge, hybridization)
- Bond feature arrays (bond type, aromaticity)
- SMILES or MOL format structure files from in-house database
Outputs
- Serialized molecular graphs in pickle format
- Serialized molecular graphs in HDF5 format
- PyTorch-compatible graph batches ready for DataLoader consumption
How to apply
After converting molecular structures into node-edge representations using RDKit's graph construction methods, serialize the molecular graphs into pickle or HDF5 format to create a format compatible with PyTorch's DataLoader. The serialized graphs should retain all atom and bond feature attributes computed during the RDKit preprocessing stage. Store serialized graphs in a structured directory (e.g., the 'data' directory) that the PyTorch training pipeline can discover and batch-load during Train.py and Transferlearning.py execution. Verify serialization by confirming file sizes are reasonable relative to the number of molecules and that DataLoader can iterate over batches without deserialization errors.
Related tools
- RDKit (Construct molecular graphs with atom/bond features before serialization) — https://www.rdkit.org/
- PyTorch (Load and batch-iterate over serialized graphs via DataLoader) — https://pytorch.org/
- Python pickle module (Serialize graph objects into binary format)
- HDF5 (h5py) (Alternative serialization format for large-scale graph storage)
Examples
python Preprocess.py # converts SMILES/MOL files to serialized molecular graphs in data/ directory for consumption by PyTorch DataLoader in Train.py
Evaluation signals
- Serialized files are created and readable by Python pickle or h5py without errors
- PyTorch DataLoader successfully iterates over batches of deserialized graphs without shape or type mismatches
- Graph features (atom features, bond features) are preserved after deserialization and match the original RDKit-constructed representations
- Training pipeline (Train.py, Transferlearning.py) runs without I/O bottlenecks or out-of-memory errors when loading batches
- File sizes scale linearly with the number of molecules and average molecular complexity
Limitations
- Serialization format choice (pickle vs HDF5) affects compatibility across Python versions and platforms; pickle is Python-specific
- Large molecular datasets may require substantial disk space; pre-serialization requires estimating storage and I/O bandwidth
- Graph serialization assumes fixed schema; heterogeneous graph structures or variable feature dimensions may require custom serialization logic
- No changelog or versioning support noted in repository, making it unclear how serialized graphs from older versions interact with updated GNN-RT code
Evidence
- [other] Serialize the molecular graphs into a format compatible with PyTorch's DataLoader (e.g., pickle or HDF5) for consumption by the GNN training pipeline.: "Serialize the molecular graphs into a format compatible with PyTorch's DataLoader (e.g., pickle or HDF5)"
- [other] Construct molecular graphs by converting each structure into a node-edge representation with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type, aromaticity) using RDKit's graph construction methods.: "Construct molecular graphs by converting each structure into a node-edge representation with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type,"
- [readme] It takes molecular graph as the input, and the predicted retention time as the output.: "It takes molecular graph as the input, and the predicted retention time as the output."
- [readme] put your spectra files in to data directory and run [Preprocess.py], [Train.py] and [Transferlearning.py]: "put your spectra files in to data directory and run [Preprocess.py], [Train.py] and [Transferlearning.py]"
1---2name: pytorch-graph-serialization3description: Use when you have constructed molecular graphs with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type, aromaticity) from SMILES or MOL files using RDKit, and need to feed them into a PyTorch-based GNN training pipeline without memory overhead or I/O.4license: CC-BY-4.05---67# PyTorch Graph Serialization for GNN Input89## Summary1011Serialize molecular graph representations into PyTorch-compatible formats (pickle or HDF5) to enable efficient consumption by Graph Neural Network training pipelines. This skill bridges the gap between RDKit molecular graph construction and PyTorch DataLoader ingestion for retention time prediction models.1213## When to use1415You have constructed molecular graphs with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type, aromaticity) from SMILES or MOL files using RDKit, and need to feed them into a PyTorch-based GNN training pipeline without memory overhead or I/O bottlenecks during model training and transfer learning phases.1617## When NOT to use1819- Input is already in PyTorch Tensor or graph object format (Data or HeteroData from PyG)20- Molecular graphs are extremely large (>10 million atoms per structure) and serialization overhead exceeds memory savings21- Training pipeline expects streaming/online data augmentation rather than pre-serialized static batches2223## Inputs2425- Molecular graphs with node-edge representations (constructed via RDKit)26- Atom feature arrays (atomic number, degree, formal charge, hybridization)27- Bond feature arrays (bond type, aromaticity)28- SMILES or MOL format structure files from in-house database2930## Outputs3132- Serialized molecular graphs in pickle format33- Serialized molecular graphs in HDF5 format34- PyTorch-compatible graph batches ready for DataLoader consumption3536## How to apply3738After converting molecular structures into node-edge representations using RDKit's graph construction methods, serialize the molecular graphs into pickle or HDF5 format to create a format compatible with PyTorch's DataLoader. The serialized graphs should retain all atom and bond feature attributes computed during the RDKit preprocessing stage. Store serialized graphs in a structured directory (e.g., the 'data' directory) that the PyTorch training pipeline can discover and batch-load during Train.py and Transferlearning.py execution. Verify serialization by confirming file sizes are reasonable relative to the number of molecules and that DataLoader can iterate over batches without deserialization errors.3940## Related tools4142- **RDKit** (Construct molecular graphs with atom/bond features before serialization) — https://www.rdkit.org/43- **PyTorch** (Load and batch-iterate over serialized graphs via DataLoader) — https://pytorch.org/44- **Python pickle module** (Serialize graph objects into binary format)45- **HDF5 (h5py)** (Alternative serialization format for large-scale graph storage)4647## Examples4849```50python Preprocess.py # converts SMILES/MOL files to serialized molecular graphs in data/ directory for consumption by PyTorch DataLoader in Train.py51```5253## Evaluation signals5455- Serialized files are created and readable by Python pickle or h5py without errors56- PyTorch DataLoader successfully iterates over batches of deserialized graphs without shape or type mismatches57- Graph features (atom features, bond features) are preserved after deserialization and match the original RDKit-constructed representations58- Training pipeline (Train.py, Transferlearning.py) runs without I/O bottlenecks or out-of-memory errors when loading batches59- File sizes scale linearly with the number of molecules and average molecular complexity6061## Limitations6263- Serialization format choice (pickle vs HDF5) affects compatibility across Python versions and platforms; pickle is Python-specific64- Large molecular datasets may require substantial disk space; pre-serialization requires estimating storage and I/O bandwidth65- Graph serialization assumes fixed schema; heterogeneous graph structures or variable feature dimensions may require custom serialization logic66- No changelog or versioning support noted in repository, making it unclear how serialized graphs from older versions interact with updated GNN-RT code6768## Evidence6970- [other] Serialize the molecular graphs into a format compatible with PyTorch's DataLoader (e.g., pickle or HDF5) for consumption by the GNN training pipeline.: "Serialize the molecular graphs into a format compatible with PyTorch's DataLoader (e.g., pickle or HDF5)"71- [other] Construct molecular graphs by converting each structure into a node-edge representation with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type, aromaticity) using RDKit's graph construction methods.: "Construct molecular graphs by converting each structure into a node-edge representation with atom features (atomic number, degree, formal charge, hybridization) and bond features (bond type,"72- [readme] It takes molecular graph as the input, and the predicted retention time as the output.: "It takes molecular graph as the input, and the predicted retention time as the output."73- [readme] put your spectra files in to data directory and run [Preprocess.py], [Train.py] and [Transferlearning.py]: "put your spectra files in to data directory and run [Preprocess.py], [Train.py] and [Transferlearning.py]"
HolobiomicsLab