# Training Dataset Assembly

> Use when when you have a large list of SMILES strings but lack corresponding experimental mass spectra, and you need to generate 30,000+ training pairs to train a deep learning model for molecular structure prediction.

- Skill: `holobiomicslab/training-dataset-assembly` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/training-dataset-assembly`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/training-dataset-assembly/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/training-dataset-assembly

---


# training-dataset-assembly

## Summary

Assemble large-scale pseudo SMILES-spectrum pairs from raw SMILES lists using in silico fragmentation prediction (CFM-ID) to create training data for molecular structure generation models. This skill is essential when labeled experimental spectra are limited but raw chemical structure databases are available.

## When to use

When you have a large list of SMILES strings but lack corresponding experimental mass spectra, and you need to generate 30,000+ training pairs to train a deep learning model for molecular structure prediction. Particularly relevant when real experimental spectra are expensive, time-consuming to acquire, or limited in diversity (e.g., <300 real spectra available for model validation).

## When NOT to use

- When you already have 300+ high-quality experimental spectra; real spectra should be prioritized for validation and serve as ground truth rather than replaced by pseudo-data.
- When the chemical space of your SMILES list is drastically different from your evaluation spectra (e.g., training on general organic molecules but evaluating on PFAS or lipids); pseudo-data generation will not bridge this domain gap.
- When CFM-ID parameters (ionization mode, collision energy) do not match the experimental conditions of your evaluation dataset; the pseudo spectra will be misaligned with real data.

## Inputs

- Raw SMILES list file (text or CSV with SMILES strings, one per line)
- CFM-ID parameter configuration (ionization mode, collision energy, fragmentation rules)

## Outputs

- Aggregated pseudo SMILES-spectrum pair dataset (≥30,000 pairs)
- Dataset file format compatible with PyTorch DataLoader (e.g., .pkl or .csv with SMILES and spectrum columns)
- Dataset validation report (pair count, schema integrity checks)

## How to apply

Load raw SMILES strings from an input file and invoke CFM-ID to generate in silico mass spectra with fragmentation patterns for each structure. Pair each input SMILES with its CFM-ID-generated spectrum to create pseudo SMILES-spectrum pairs. Aggregate all pairs into a single training dataset file, accumulating at least 30,000+ pairs to provide sufficient diversity and scale for deep learning. Validate the output dataset by confirming: (1) the file contains the expected number of pairs (≥30k), (2) each pair includes both a valid SMILES string and a corresponding spectrum representation (format suitable for torch.nn input), and (3) spectrum distributions match expected fragmentation characteristics. Use Python 3.7 with PyTorch 1.7.1 for downstream model compatibility.

## Related tools

- **CFM-ID** (In silico mass spectra generation from SMILES; produces fragmentation patterns and spectrum predictions for each molecular structure)
- **Python** (Orchestration and aggregation; iterate over SMILES list, invoke CFM-ID, pair results, validate output dataset)
- **PyTorch** (Data format compatibility and downstream model training; ensures pseudo pairs can be loaded into MSGO or similar neural network architectures) — https://github.com/aaronma2020/MSGO

## Examples

```
# Load raw SMILES, generate spectra with CFM-ID, and aggregate pairs:
for smiles in open('raw_smiles.txt'):
    spectrum = cfmid_predict(smiles.strip(), ionization='neg')
    pseudo_pairs.append({'SMILES': smiles.strip(), 'spectrum': spectrum})
import pickle
with open('training_data_30k.pkl', 'wb') as f:
    pickle.dump(pseudo_pairs, f)
print(f'Generated {len(pseudo_pairs)} pseudo SMILES-spectrum pairs')
```

## Evaluation signals

- Output dataset file contains ≥30,000 SMILES-spectrum pairs with no null entries or malformed SMILES strings.
- Each spectrum entry is a valid numeric array or tensor representation compatible with torch.nn models, with consistent dimensionality across all pairs.
- SMILES strings in output pairs are identical to input SMILES strings (verify by checksum or line-by-line comparison).
- Spectrum fragmentation patterns follow expected mass-to-charge (m/z) distributions and peak intensities consistent with CFM-ID predicted fragmentation rules for the corresponding molecular structures.
- Dataset file can be successfully loaded and iterated by a PyTorch DataLoader without shape or type errors when used in the training pipeline (test with `python tools/train.py --id all_trick`).

## Limitations

- CFM-ID-generated spectra are in silico predictions and may not capture all real-world fragmentation complexity, especially for novel or unusual structural motifs. Pseudo-data is best used for pre-training or data augmentation, not as sole training source.
- Assembly quality depends critically on CFM-ID parameter choices (ionization mode, collision energy); mismatch between pseudo-data generation settings and evaluation dataset experimental conditions will degrade model performance.
- Requires valid SMILES strings as input; malformed or invalid SMILES will cause CFM-ID to fail or produce empty spectra. Pre-validation of input SMILES is recommended.
- Computational cost scales linearly with number of SMILES; generating 30,000+ spectra via CFM-ID can be time-intensive; parallelization or batch processing may be necessary for large-scale datasets.
- The 30,000+ threshold is empirical for MSGO; datasets substantially smaller may underprovide coverage of chemical space and lead to overfitting or poor generalization.

## Evidence

- [other] Training data for MSGO consists of 30k+ pseudo SMILES-spectrum pairs generated by CFM-ID from raw SMILES lists.: "Training data for MSGO consists of 30k+ pseudo SMILES-spectrum pairs generated by CFM-ID from raw SMILES lists."
- [readme] For Training, we use 30k+ pseudo smiles-specturm pairs generated by cfmid: "For Training, we use 30k+ pseudo smiles-specturm pairs generated by cfmid"
- [other] For each SMILES string, invoke CFM-ID to generate in silico mass spectra with fragmentation patterns. Pair each input SMILES with its corresponding CFM-ID-generated spectrum to create pseudo SMILES-spectrum pairs.: "For each SMILES string, invoke CFM-ID to generate in silico mass spectra with fragmentation patterns. Pair each input SMILES with its corresponding CFM-ID-generated spectrum to create pseudo"
- [readme] For evaluation, we use 300+ real specturm to verify our method: "For evaluation, we use 300+ real specturm to verify our method"
- [readme] Python: 3.7 \ Torch: 1.7.1: "Python: 3.7 \ Torch: 1.7.1"