spectral-peak-filtering-and-normalization
Summary
Preprocessing pipeline that removes low-intensity peaks, normalizes peak intensities via square-root transformation, and bins peaks into fixed m/z ranges to prepare raw MS/MS spectra for machine learning. This standardization ensures consistent input representation across spectra of varying quality and intensity distributions.
When to use
Apply this skill when working with raw or minimally processed MS/MS spectra from repositories like GNPS that contain variable peak intensities, noise, and formatting inconsistencies that would interfere with downstream deep learning models trained on normalized spectral representations. Specifically useful when the goal is to train or apply neural networks that expect fixed-dimension spectral vectors rather than variable-length peak lists.
When NOT to use
- Input spectra are already binned or feature-engineered into fixed vectors — reapplying binning will lose information or double-bin the data.
- Analysis requires preservation of exact peak positions (e.g., high-resolution accurate mass workflows or isotope pattern analysis) — binning to 10,000 bins (0.1 m/z resolution) is too coarse.
- Downstream model explicitly uses raw peak intensities as training data — the square-root transformation may not be compatible with models trained on untransformed intensity distributions.
Inputs
- raw MS/MS spectra (mzML, mgf, msp, or json format)
- peak lists with m/z and intensity pairs
- mass range bounds (default 10–1000 m/z)
Outputs
- binned spectral vectors (10,000-element arrays, one per spectrum)
- filtered and normalized peak intensity values
- metadata indicating number of peaks retained and bins populated
How to apply
Remove peaks with intensities below 0.1% of the maximum peak intensity within each spectrum, then cap the spectrum to its 1000 highest-intensity peaks to reduce noise and computational burden. Apply a square-root transformation to all remaining peak intensities to reduce the dominance of very high peaks and normalize the intensity scale. Finally, bin the transformed peaks into 10,000 equally-sized bins spanning 10–1000 m/z (0.1 m/z per bin), selecting the maximum intensity when multiple peaks fall into the same bin. This produces a fixed-length vector representation suitable for Siamese neural network training. The square-root transformation is critical because it avoids overfitting to the highest intensity peaks, which often dominate MS/MS spectra but may not be diagnostically informative for structural similarity prediction.
Related tools
- matchms (Metadata cleaning and standardization prior to peak filtering; handles compound name normalization, adduct extraction, and InChI/SMILES conversion) — https://github.com/matchms/matchms
- RDKit (Molecular fingerprint generation from chemical structure annotations to compute ground-truth structural similarity labels for training data validation)
- Python (Implementation language for peak filtering, intensity transformation, and binning operations)
Examples
from matchms.importing_utils import load_documents; from matchms import Spectrum; spectra = load_documents('spectra.mgf'); filtered_spectra = [s for s in spectra if len(s.peaks) >= 5]; binned = [[max([p[1] for p in s.peaks if 10 <= (p[0]//0.1)*0.1 < 1000 and (p[0]//0.1)*0.1 == b*0.1], default=0)**0.5 for b in range(10000)] for s in filtered_spectra]
Evaluation signals
- Verify that all output spectral vectors have exactly 10,000 bins with no missing dimensions.
- Confirm that no peak intensities exceed 1.0 (the maximum after square-root transformation of normalized intensities); spot-check a sample of transformed values.
- Check that the number of non-zero bins per spectrum is consistent with expected diversity (most spectra should have 50–500 populated bins; very sparse spectra (<10 bins) or fully populated spectra (>9900 bins) may indicate pipeline errors).
- Validate that peak filtering removed ≥90% of the original low-intensity noise by comparing median peak count before and after filtering (raw spectra typically have 100–500 peaks; after filtering and binning, 50–500 should remain).
- Ensure that 9,948 of 10,000 bins contain at least one spectrum's peak intensity in the training set, confirming the binning is not over-sparse.
Limitations
- Fixed 10,000-bin architecture (0.1 m/z per bin) may lose information in high-resolution accurate mass spectra or miss fine isotope patterns relevant to structure elucidation.
- Square-root transformation assumes intensity follows a scale-dependent noise model; if peak intensity noise is additive (not multiplicative), this transformation may be suboptimal.
- Discarding the top 1000 peaks and removing intensities <0.1% of max may remove diagnostic low-intensity fragments in spectra where weak but structurally informative peaks are present.
- The pipeline does not account for systematic instrumental variations (e.g., different ionization methods, collision energies, instrument types) and treats all spectra uniformly; cross-ionization or cross-instrument predictions may suffer.
- Peak binning with maximum-intensity selection discards information about peak multiplicity within bins; if two isobars of different intensities occupy the same bin, only the higher is retained.
Evidence
- [methods] by removing peaks with intensities < 0.1% of the maximum peak intensity and limiting the maximum number of peaks to the 1000 highest intensity peaks: "by removing peaks with intensities < 0.1% of the maximum peak intensity and limiting the maximum number of peaks to the 1000 highest intensity peaks"
- [methods] Peak intensities were square root transformed to avoid a too strong focus on the highest intensity peaks only: "Peak intensities were square root transformed to avoid a too strong focus on the highest intensity peaks only"
- [methods] Spectrum peaks were binned in 10,000 equally-sized bins ranging from 10 to 1000 m/z: "Spectrum peaks were binned in 10,000 equally-sized bins ranging from 10 to 1000 m/z"
- [other] Filter spectra to retain only those in positive ionization mode with valid 14-character InChIKey and SMILES/InChI annotation, containing ≥5 peaks in the 10.0–1000.0 Da mass range: "Filter spectra to retain only those in positive ionization mode with valid 14-character InChIKey and SMILES/InChI annotation, containing ≥5 peaks in the 10.0–1000.0 Da mass range"
- [other] Remove bins not represented in training data (9948 of 10,000 bins retained) and output cleaned spectral vectors and metadata: "Remove bins not represented in training data (9948 of 10,000 bins retained) and output cleaned spectral vectors and metadata"
1---2name: spectral-peak-filtering-and-normalization3description: Use when working with raw or minimally processed MS/MS spectra from repositories like GNPS that contain variable peak intensities, noise, and formatting inconsistencies that would interfere with downstream deep learning models trained on normalized spectral representations.4license: CC-BY-4.05---67# spectral-peak-filtering-and-normalization89## Summary1011Preprocessing pipeline that removes low-intensity peaks, normalizes peak intensities via square-root transformation, and bins peaks into fixed m/z ranges to prepare raw MS/MS spectra for machine learning. This standardization ensures consistent input representation across spectra of varying quality and intensity distributions.1213## When to use1415Apply this skill when working with raw or minimally processed MS/MS spectra from repositories like GNPS that contain variable peak intensities, noise, and formatting inconsistencies that would interfere with downstream deep learning models trained on normalized spectral representations. Specifically useful when the goal is to train or apply neural networks that expect fixed-dimension spectral vectors rather than variable-length peak lists.1617## When NOT to use1819- Input spectra are already binned or feature-engineered into fixed vectors — reapplying binning will lose information or double-bin the data.20- Analysis requires preservation of exact peak positions (e.g., high-resolution accurate mass workflows or isotope pattern analysis) — binning to 10,000 bins (0.1 m/z resolution) is too coarse.21- Downstream model explicitly uses raw peak intensities as training data — the square-root transformation may not be compatible with models trained on untransformed intensity distributions.2223## Inputs2425- raw MS/MS spectra (mzML, mgf, msp, or json format)26- peak lists with m/z and intensity pairs27- mass range bounds (default 10–1000 m/z)2829## Outputs3031- binned spectral vectors (10,000-element arrays, one per spectrum)32- filtered and normalized peak intensity values33- metadata indicating number of peaks retained and bins populated3435## How to apply3637Remove peaks with intensities below 0.1% of the maximum peak intensity within each spectrum, then cap the spectrum to its 1000 highest-intensity peaks to reduce noise and computational burden. Apply a square-root transformation to all remaining peak intensities to reduce the dominance of very high peaks and normalize the intensity scale. Finally, bin the transformed peaks into 10,000 equally-sized bins spanning 10–1000 m/z (0.1 m/z per bin), selecting the maximum intensity when multiple peaks fall into the same bin. This produces a fixed-length vector representation suitable for Siamese neural network training. The square-root transformation is critical because it avoids overfitting to the highest intensity peaks, which often dominate MS/MS spectra but may not be diagnostically informative for structural similarity prediction.3839## Related tools4041- **matchms** (Metadata cleaning and standardization prior to peak filtering; handles compound name normalization, adduct extraction, and InChI/SMILES conversion) — https://github.com/matchms/matchms42- **RDKit** (Molecular fingerprint generation from chemical structure annotations to compute ground-truth structural similarity labels for training data validation)43- **Python** (Implementation language for peak filtering, intensity transformation, and binning operations)4445## Examples4647```48from matchms.importing_utils import load_documents; from matchms import Spectrum; spectra = load_documents('spectra.mgf'); filtered_spectra = [s for s in spectra if len(s.peaks) >= 5]; binned = [[max([p[1] for p in s.peaks if 10 <= (p[0]//0.1)*0.1 < 1000 and (p[0]//0.1)*0.1 == b*0.1], default=0)**0.5 for b in range(10000)] for s in filtered_spectra]49```5051## Evaluation signals5253- Verify that all output spectral vectors have exactly 10,000 bins with no missing dimensions.54- Confirm that no peak intensities exceed 1.0 (the maximum after square-root transformation of normalized intensities); spot-check a sample of transformed values.55- Check that the number of non-zero bins per spectrum is consistent with expected diversity (most spectra should have 50–500 populated bins; very sparse spectra (<10 bins) or fully populated spectra (>9900 bins) may indicate pipeline errors).56- Validate that peak filtering removed ≥90% of the original low-intensity noise by comparing median peak count before and after filtering (raw spectra typically have 100–500 peaks; after filtering and binning, 50–500 should remain).57- Ensure that 9,948 of 10,000 bins contain at least one spectrum's peak intensity in the training set, confirming the binning is not over-sparse.5859## Limitations6061- Fixed 10,000-bin architecture (0.1 m/z per bin) may lose information in high-resolution accurate mass spectra or miss fine isotope patterns relevant to structure elucidation.62- Square-root transformation assumes intensity follows a scale-dependent noise model; if peak intensity noise is additive (not multiplicative), this transformation may be suboptimal.63- Discarding the top 1000 peaks and removing intensities <0.1% of max may remove diagnostic low-intensity fragments in spectra where weak but structurally informative peaks are present.64- The pipeline does not account for systematic instrumental variations (e.g., different ionization methods, collision energies, instrument types) and treats all spectra uniformly; cross-ionization or cross-instrument predictions may suffer.65- Peak binning with maximum-intensity selection discards information about peak multiplicity within bins; if two isobars of different intensities occupy the same bin, only the higher is retained.6667## Evidence6869- [methods] by removing peaks with intensities < 0.1% of the maximum peak intensity and limiting the maximum number of peaks to the 1000 highest intensity peaks: "by removing peaks with intensities < 0.1% of the maximum peak intensity and limiting the maximum number of peaks to the 1000 highest intensity peaks"70- [methods] Peak intensities were square root transformed to avoid a too strong focus on the highest intensity peaks only: "Peak intensities were square root transformed to avoid a too strong focus on the highest intensity peaks only"71- [methods] Spectrum peaks were binned in 10,000 equally-sized bins ranging from 10 to 1000 m/z: "Spectrum peaks were binned in 10,000 equally-sized bins ranging from 10 to 1000 m/z"72- [other] Filter spectra to retain only those in positive ionization mode with valid 14-character InChIKey and SMILES/InChI annotation, containing ≥5 peaks in the 10.0–1000.0 Da mass range: "Filter spectra to retain only those in positive ionization mode with valid 14-character InChIKey and SMILES/InChI annotation, containing ≥5 peaks in the 10.0–1000.0 Da mass range"73- [other] Remove bins not represented in training data (9948 of 10,000 bins retained) and output cleaned spectral vectors and metadata: "Remove bins not represented in training data (9948 of 10,000 bins retained) and output cleaned spectral vectors and metadata"
HolobiomicsLab