mass-spectrometry-dataset-handling-preprocessing
Summary
Load, parse, and preprocess tandem mass spectrometry (MS/MS) data from MSP format files into structured arrays suitable for neural embedding models. This skill bridges raw spectral data and machine learning pipelines by standardizing spectra representation, extracting chemical metadata (SMILES), and preparing batches for model inference.
When to use
You have raw or semi-processed MS/MS spectra in MSP format (e.g., from GNPS, Orbitrap instruments) and need to feed them into MSBERT or similar transformer-based embedding models for library matching, clustering, or similarity scoring. The input spectra may contain variable peak intensities, precursor m/z values, and SMILES annotations that must be normalized and aligned to a fixed vocabulary size (e.g., 100,002 m/z bins).
When NOT to use
- Input is already a pre-computed embedding matrix or feature table; skip directly to similarity/matching steps.
- Spectra are in non-MSP formats (raw mzML, mzXML, raw proprietary formats) without prior conversion to MSP.
- SMILES annotations are absent and structural ground truth is required for evaluation; consider retrieving SMILES from external databases first.
Inputs
- MSP format spectral files (.msp)
- Spectra with peaks (m/z, intensity pairs)
- SMILES strings or chemical structure annotations
- Pre-trained MSBERT model state dictionary (.pkl)
Outputs
- Preprocessed spectral numpy arrays or tensors
- MSBERT embedding vectors (shape: num_spectra × 512)
- SMILES metadata aligned to spectra
- Cosine similarity matrices (when MSBERTSimilarity is applied)
How to apply
Parse MSP files using the ProcessMSP utility to extract spectral peak arrays and corresponding SMILES strings. Convert each spectrum into a fixed-length numerical representation (e.g., binned m/z intensities over a defined mass range with vocabulary size 100,002) aligned to the model's input embedding dimension. Pass the preprocessed spectral arrays through ModelEmbed to generate embeddings at a specified batch size (e.g., 16 spectra per batch). Validate that output embedding vectors match the model's latent dimension (e.g., 512 dimensions for MSBERT) and that no spectra were dropped or corrupted during the conversion. Store embeddings and retain SMILES metadata for downstream similarity calculations or validation against structural ground truth.
Related tools
Examples
import pickle
import torch
from model.utils import ModelEmbed, ProcessMSP
model_file = 'model/MSBERT.pkl'
model = torch.load(model_file)
demo_data, demo_smiles = ProcessMSP('example/demo.msp')
demo_arr = ModelEmbed(model, demo_data, 16)
Evaluation signals
- Preprocessed spectral arrays have shape (num_spectra, vocab_size=100,002) with non-negative intensity values summing to expected norms per spectrum.
- MSBERT embeddings have exact shape (num_spectra, 512) with no NaN or Inf values; all tensors are on the same device (CPU/GPU).
- SMILES metadata count matches preprocessed spectra count; no spectra are dropped or duplicated during conversion.
- Cosine similarity scores computed via MSBERTSimilarity fall in [−1, 1] range; diagonal values (self-similarity) are ≥0.99, confirming embedding stability.
- Reprocessed demo.msp spectra produce identical embeddings to those in the provided example, confirming pipeline reproducibility.
Limitations
- Requires MSP format input; other spectral formats (mzML, mzXML, HDF5) must be pre-converted to MSP.
- Fixed vocabulary size (100,002 m/z bins) is tuned to GNPS Orbitrap data; retraining on different mass ranges or instruments may require vocabulary re-calibration.
- SMILES annotations are optional for embedding but required for downstream structural similarity validation; spectra without SMILES will pass preprocessing but cannot be validated against chemical ground truth.
- Batch size (e.g., 16) must fit in available GPU/CPU memory; very large spectra batches or high embedding dimensions may require smaller batch sizes or data sharding.
Evidence
- [readme] demo_data,demo_smiles = ProcessMSP(demo_file): "demo_file = 'example/demo.msp'
demo_data,demo_smiles = ProcessMSP(demo_file)"
- [readme] demo_arr = ModelEmbed(model,demo_data,16): "demo_arr = ModelEmbed(model,demo_data,16)"
- [readme] MSBERT(100002, 512, 6, 16, 0,100,3): "model = MSBERT(100002, 512, 6, 16, 0,100,3)"
- [intro] Load the pre-trained MSBERT model and the GNPS-derived Orbitrap test dataset.: "Load the pre-trained MSBERT model and the GNPS-derived Orbitrap test dataset."
- [other] MSBERT was trained and tested on GNPS dataset.: "MSBERT was trained and tested on GNPS dataset."
- [readme] The filtered GNPS dataset used in this experiment was uploaded to Zenodo: "The filtered GNPS dataset used in this experiment was uploaded to Zenodo."
1---2name: mass-spectrometry-dataset-handling-preprocessing-23description: Use when you have raw or semi-processed MS/MS spectra in MSP format (e.g.,4 from GNPS, Orbitrap instruments) and need to feed them into MSBERT or similar transformer-based5 embedding models for library matching, clustering, or similarity scoring.6license: CC-BY-4.07metadata:8 source: https://skillmd.com/skills/holobiomicslab/mass-spectrometry-dataset-handling-preprocessing-29 edam_operation: http://edamontology.org/operation_380210 edam_topics:11 - http://edamontology.org/topic_012112 - http://edamontology.org/topic_352013 tools:14 - Python15 - Anaconda16 - Git17 - ProcessMSP18 - ModelEmbed19 - MakeTrainData20 - PyTorch 2.221 - Python 3.12 with Anaconda22 techniques:23 - LC-MS24 license_tier: open25 provenance_tier: literature26derived_from:27- doi: 10.1021/acs.analchem.4c0242628 title: MSBERT29evidence_spans:30- '[Anaconda](https://www.anaconda.com) for Python 3.12'31- Install [Git](https://git-scm.com/downloads)32claims: []33provenance:34 collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v235 assembled_by: scripts/collect_metabolomics_collection.py36 sources:37 - build: coll_msbert_cq38 doi: 10.1021/acs.analchem.4c0242639 title: MSBERT40 dedup_kept_from: coll_msbert_cq41schema_version: 0.2.042attribution:43 generator: AgenticScienceBuilder44 original_doi: 10.1021/acs.analchem.4c0242645 all_source_dois:46 - 10.1021/acs.analchem.4c0242647 zenodo_doi: 10.5281/zenodo.2079402748 curators: []49 promoter: Louis-Félix Nothias50 sponsor: CNRS & Université Côte d'Azur51---5253# mass-spectrometry-dataset-handling-preprocessing5455## Summary5657Load, parse, and preprocess tandem mass spectrometry (MS/MS) data from MSP format files into structured arrays suitable for neural embedding models. This skill bridges raw spectral data and machine learning pipelines by standardizing spectra representation, extracting chemical metadata (SMILES), and preparing batches for model inference.5859## When to use6061You have raw or semi-processed MS/MS spectra in MSP format (e.g., from GNPS, Orbitrap instruments) and need to feed them into MSBERT or similar transformer-based embedding models for library matching, clustering, or similarity scoring. The input spectra may contain variable peak intensities, precursor m/z values, and SMILES annotations that must be normalized and aligned to a fixed vocabulary size (e.g., 100,002 m/z bins).6263## When NOT to use6465- Input is already a pre-computed embedding matrix or feature table; skip directly to similarity/matching steps.66- Spectra are in non-MSP formats (raw mzML, mzXML, raw proprietary formats) without prior conversion to MSP.67- SMILES annotations are absent and structural ground truth is required for evaluation; consider retrieving SMILES from external databases first.6869## Inputs7071- MSP format spectral files (.msp)72- Spectra with peaks (m/z, intensity pairs)73- SMILES strings or chemical structure annotations74- Pre-trained MSBERT model state dictionary (.pkl)7576## Outputs7778- Preprocessed spectral numpy arrays or tensors79- MSBERT embedding vectors (shape: num_spectra × 512)80- SMILES metadata aligned to spectra81- Cosine similarity matrices (when MSBERTSimilarity is applied)8283## How to apply8485Parse MSP files using the ProcessMSP utility to extract spectral peak arrays and corresponding SMILES strings. Convert each spectrum into a fixed-length numerical representation (e.g., binned m/z intensities over a defined mass range with vocabulary size 100,002) aligned to the model's input embedding dimension. Pass the preprocessed spectral arrays through ModelEmbed to generate embeddings at a specified batch size (e.g., 16 spectra per batch). Validate that output embedding vectors match the model's latent dimension (e.g., 512 dimensions for MSBERT) and that no spectra were dropped or corrupted during the conversion. Store embeddings and retain SMILES metadata for downstream similarity calculations or validation against structural ground truth.8687## Related tools8889- **ProcessMSP** (Parse MSP files and extract spectral peaks and SMILES; foundational data loading utility) — https://github.com/zhanghailiangcsu/MSBERT90- **ModelEmbed** (Convert preprocessed spectral arrays into fixed-dimension MSBERT embedding vectors at specified batch size) — https://github.com/zhanghailiangcsu/MSBERT91- **MakeTrainData** (Prepare and augment training datasets from raw GNPS data; used when retraining on custom datasets) — https://github.com/zhanghailiangcsu/MSBERT92- **PyTorch 2.2** (Tensor computation and model loading backend for embedding generation) — https://pytorch.org/93- **Python 3.12 with Anaconda** (Runtime environment and dependency management) — https://www.anaconda.com9495## Examples9697```98import pickle99import torch100from model.utils import ModelEmbed, ProcessMSP101model_file = 'model/MSBERT.pkl'102model = torch.load(model_file)103demo_data, demo_smiles = ProcessMSP('example/demo.msp')104demo_arr = ModelEmbed(model, demo_data, 16)105```106107## Evaluation signals108109- Preprocessed spectral arrays have shape (num_spectra, vocab_size=100,002) with non-negative intensity values summing to expected norms per spectrum.110- MSBERT embeddings have exact shape (num_spectra, 512) with no NaN or Inf values; all tensors are on the same device (CPU/GPU).111- SMILES metadata count matches preprocessed spectra count; no spectra are dropped or duplicated during conversion.112- Cosine similarity scores computed via MSBERTSimilarity fall in [−1, 1] range; diagonal values (self-similarity) are ≥0.99, confirming embedding stability.113- Reprocessed demo.msp spectra produce identical embeddings to those in the provided example, confirming pipeline reproducibility.114115## Limitations116117- Requires MSP format input; other spectral formats (mzML, mzXML, HDF5) must be pre-converted to MSP.118- Fixed vocabulary size (100,002 m/z bins) is tuned to GNPS Orbitrap data; retraining on different mass ranges or instruments may require vocabulary re-calibration.119- SMILES annotations are optional for embedding but required for downstream structural similarity validation; spectra without SMILES will pass preprocessing but cannot be validated against chemical ground truth.120- Batch size (e.g., 16) must fit in available GPU/CPU memory; very large spectra batches or high embedding dimensions may require smaller batch sizes or data sharding.121122## Evidence123124- [readme] demo_data,demo_smiles = ProcessMSP(demo_file): "demo_file = 'example/demo.msp'125demo_data,demo_smiles = ProcessMSP(demo_file)"126- [readme] demo_arr = ModelEmbed(model,demo_data,16): "demo_arr = ModelEmbed(model,demo_data,16)"127- [readme] MSBERT(100002, 512, 6, 16, 0,100,3): "model = MSBERT(100002, 512, 6, 16, 0,100,3)"128- [intro] Load the pre-trained MSBERT model and the GNPS-derived Orbitrap test dataset.: "Load the pre-trained MSBERT model and the GNPS-derived Orbitrap test dataset."129- [other] MSBERT was trained and tested on GNPS dataset.: "MSBERT was trained and tested on GNPS dataset."130- [readme] The filtered GNPS dataset used in this experiment was uploaded to Zenodo: "The filtered GNPS dataset used in this experiment was uploaded to [Zenodo](https://zenodo.org/records/13722644)."