Training Dataset Assembly

training-dataset-assembly

Summary

Assemble large-scale pseudo SMILES-spectrum pairs from raw SMILES lists using in silico fragmentation prediction (CFM-ID) to create training data for molecular structure generation models. This skill is essential when labeled experimental spectra are limited but raw chemical structure databases are available.

When to use

When you have a large list of SMILES strings but lack corresponding experimental mass spectra, and you need to generate 30,000+ training pairs to train a deep learning model for molecular structure prediction. Particularly relevant when real experimental spectra are expensive, time-consuming to acquire, or limited in diversity (e.g., <300 real spectra available for model validation).

When NOT to use

• When you already have 300+ high-quality experimental spectra; real spectra should be prioritized for validation and serve as ground truth rather than replaced by pseudo-data.
• When the chemical space of your SMILES list is drastically different from your evaluation spectra (e.g., training on general organic molecules but evaluating on PFAS or lipids); pseudo-data generation will not bridge this domain gap.
• When CFM-ID parameters (ionization mode, collision energy) do not match the experimental conditions of your evaluation dataset; the pseudo spectra will be misaligned with real data.

Inputs

• Raw SMILES list file (text or CSV with SMILES strings, one per line)
• CFM-ID parameter configuration (ionization mode, collision energy, fragmentation rules)

Outputs

• Aggregated pseudo SMILES-spectrum pair dataset (≥30,000 pairs)
• Dataset file format compatible with PyTorch DataLoader (e.g., .pkl or .csv with SMILES and spectrum columns)
• Dataset validation report (pair count, schema integrity checks)

How to apply

Load raw SMILES strings from an input file and invoke CFM-ID to generate in silico mass spectra with fragmentation patterns for each structure. Pair each input SMILES with its CFM-ID-generated spectrum to create pseudo SMILES-spectrum pairs. Aggregate all pairs into a single training dataset file, accumulating at least 30,000+ pairs to provide sufficient diversity and scale for deep learning. Validate the output dataset by confirming: (1) the file contains the expected number of pairs (≥30k), (2) each pair includes both a valid SMILES string and a corresponding spectrum representation (format suitable for torch.nn input), and (3) spectrum distributions match expected fragmentation characteristics. Use Python 3.7 with PyTorch 1.7.1 for downstream model compatibility.

Related tools

• CFM-ID (In silico mass spectra generation from SMILES; produces fragmentation patterns and spectrum predictions for each molecular structure)
• Python (Orchestration and aggregation; iterate over SMILES list, invoke CFM-ID, pair results, validate output dataset)
• PyTorch (Data format compatibility and downstream model training; ensures pseudo pairs can be loaded into MSGO or similar neural network architectures) — https://github.com/aaronma2020/MSGO

Examples

# Load raw SMILES, generate spectra with CFM-ID, and aggregate pairs:
for smiles in open('raw_smiles.txt'):
    spectrum = cfmid_predict(smiles.strip(), ionization='neg')
    pseudo_pairs.append({'SMILES': smiles.strip(), 'spectrum': spectrum})
import pickle
with open('training_data_30k.pkl', 'wb') as f:
    pickle.dump(pseudo_pairs, f)
print(f'Generated {len(pseudo_pairs)} pseudo SMILES-spectrum pairs')

Evaluation signals

• Output dataset file contains ≥30,000 SMILES-spectrum pairs with no null entries or malformed SMILES strings.
• Each spectrum entry is a valid numeric array or tensor representation compatible with torch.nn models, with consistent dimensionality across all pairs.
• SMILES strings in output pairs are identical to input SMILES strings (verify by checksum or line-by-line comparison).
• Spectrum fragmentation patterns follow expected mass-to-charge (m/z) distributions and peak intensities consistent with CFM-ID predicted fragmentation rules for the corresponding molecular structures.
• Dataset file can be successfully loaded and iterated by a PyTorch DataLoader without shape or type errors when used in the training pipeline (test with python tools/train.py --id all_trick).

Limitations

• CFM-ID-generated spectra are in silico predictions and may not capture all real-world fragmentation complexity, especially for novel or unusual structural motifs. Pseudo-data is best used for pre-training or data augmentation, not as sole training source.
• Assembly quality depends critically on CFM-ID parameter choices (ionization mode, collision energy); mismatch between pseudo-data generation settings and evaluation dataset experimental conditions will degrade model performance.
• Requires valid SMILES strings as input; malformed or invalid SMILES will cause CFM-ID to fail or produce empty spectra. Pre-validation of input SMILES is recommended.
• Computational cost scales linearly with number of SMILES; generating 30,000+ spectra via CFM-ID can be time-intensive; parallelization or batch processing may be necessary for large-scale datasets.
• The 30,000+ threshold is empirical for MSGO; datasets substantially smaller may underprovide coverage of chemical space and lead to overfitting or poor generalization.

Evidence

• [other] Training data for MSGO consists of 30k+ pseudo SMILES-spectrum pairs generated by CFM-ID from raw SMILES lists.: "Training data for MSGO consists of 30k+ pseudo SMILES-spectrum pairs generated by CFM-ID from raw SMILES lists."
• [readme] For Training, we use 30k+ pseudo smiles-specturm pairs generated by cfmid: "For Training, we use 30k+ pseudo smiles-specturm pairs generated by cfmid"
• [other] For each SMILES string, invoke CFM-ID to generate in silico mass spectra with fragmentation patterns. Pair each input SMILES with its corresponding CFM-ID-generated spectrum to create pseudo SMILES-spectrum pairs.: "For each SMILES string, invoke CFM-ID to generate in silico mass spectra with fragmentation patterns. Pair each input SMILES with its corresponding CFM-ID-generated spectrum to create pseudo"
• [readme] For evaluation, we use 300+ real specturm to verify our method: "For evaluation, we use 300+ real specturm to verify our method"
• [readme] Python: 3.7 \ Torch: 1.7.1: "Python: 3.7 \ Torch: 1.7.1"