high-resolution-mass-spectrometry-data-preprocessing
Summary
Transform high-resolution tandem mass spectrometry (MS/MS) spectra from mzML, mzXML, or MGF format into low-dimensional hashed feature vectors suitable for fast nearest-neighbor indexing and clustering. This preprocessing step is essential for scaling spectrum similarity searching to millions of spectra without exhaustive pairwise comparisons.
When to use
You have acquired high-resolution MS/MS spectra in mzML, mzXML, or MGF format and need to prepare them for large-scale clustering or similarity searching. Apply this skill when your analysis goal requires finding similar spectra across thousands or millions of spectra efficiently, or when you intend to construct nearest-neighbor indexes to avoid computing all pairwise distances exhaustively.
When NOT to use
- Input spectra are already in low-dimensional vector or feature table form (e.g., quantification matrices from label-free or labeled proteomics).
- Your analysis requires interpretation of individual fragment ion assignments; feature hashing obscures which specific m/z bins contributed to each hash bucket.
- You need to preserve exact mass precision for substructure annotation or metabolite identification; binning and hashing approximate similarity at the cost of exact mass fidelity.
Inputs
- mzML peak files
- mzXML peak files
- MGF peak files
Outputs
- Low-dimensional hashed feature matrix (NumPy array or SciPy sparse matrix format)
- CSV file with spectrum identifiers and corresponding feature vectors
- Filtered spectrum metadata (precursor m/z, charge state, retention time)
How to apply
Load peak files in mzML, mzXML, or MGF format using spectrum-utils or equivalent tools. Apply spectrum-level filtering (e.g., discard spectra with fewer than 5 peaks or spanning less than 250 m/z range; default min_mz 101, max_mz 500) to remove low-quality spectra before vectorization. Bin high-resolution spectra into small mass bins to create sparse, high-dimensional vectors that tightly capture fragment mass values. Apply feature hashing using a non-cryptographic hash function (e.g., MurmurHash3) to map the sparse bins into a fixed, low-dimensional vector space (controlled by the low_dim parameter). Optionally scale peak intensities (square root, logarithm, or rank scaling) to improve downstream cosine similarity estimates. Output the resulting hashed feature matrix as a dense or sparse numerical format (NumPy array, SciPy sparse matrix, or CSV). The hashing step preserves cosine similarity between the original high-resolution spectra while reducing dimensionality and memory footprint, making the vectors suitable for fast nearest-neighbor searching.
Related tools
- falcon (Full spectrum clustering pipeline that consumes the hashed feature vectors produced by this preprocessing step to perform density-based clustering via nearest-neighbor indexes) — https://github.com/bittremieux/falcon
- spectrum-utils (Utility library for reading, filtering, and normalizing MS/MS spectra from mzML, mzXML, and MGF formats prior to feature hashing)
Examples
falcon peak/*.mzml falcon --export_representatives --precursor_tol 20 ppm --fragment_tol 0.05 --eps 0.10
Evaluation signals
- Output feature vectors have fixed dimensionality equal to the
low_dim parameter (e.g., all rows in NumPy array or sparse matrix have same number of columns).
- Cosine similarity between hashed vectors approximates the true cosine similarity of the original high-resolution spectra (verify on a representative sample by comparing hashed vs. unhashed cosine scores).
- Memory footprint of the feature matrix is substantially smaller than the original mzML/mzXML files (sparse matrix representation should be ≤ 10% of raw file size for typical proteomics data).
- Filtered spectra retain expected precursor m/z and charge state distributions; no unexpected loss of spectra outside the specified min_mz and max_mz bounds.
- Downstream nearest-neighbor indexing succeeds without out-of-memory errors, and clustering results show expected cluster purity (spectra in the same cluster correspond to the same peptide or compound).
Limitations
- Feature hashing reduces interpretability: the contribution of specific fragment ions to similarity scores is obscured by the hash function, making post-hoc validation of cluster assignments difficult.
- The choice of
low_dim parameter trades off memory and speed against accuracy; smaller vectors may miss true neighbors in high-dimensional space, resulting in spurious cluster fragmentation.
- Spectrum preprocessing settings (min_peaks, min_mz_range, min_mz, max_mz, scaling method) are optimized for bottom-up proteomics by default; metabolomics or top-down data require manual adjustment of these thresholds.
- Feature hashing is sensitive to the choice of hash function and collision rate; collision-induced artifacts can accumulate if the vector dimensionality is too low relative to the number of mass bins.
- The method assumes mass spectral cosine similarity is the appropriate metric for downstream clustering; other similarity measures (e.g., spectral angle, entropy-based) are not accounted for in the hashing scheme.
Evidence
- [readme] Spectrum vectorization and hashing mechanism: "First, high-resolution spectra are binned and converted to low-dimensional vectors using feature hashing."
- [readme] Input file formats accepted: "falcon takes peak files (in the mzML, mzXML, or MGF format) as input"
- [readme] Feature hashing algorithm details: "the sparse, high-dimensional, vectors are hashed to lower-dimensional vectors by using a hash function (the non-cryptographic MurmurHash3 algorithm) to map the mass bins separately to a small number"
- [readme] Cosine similarity preservation: "This feature hashing conserves the cosine similarity between hashed vectors and can be used to approximate the similarity between the original spectra."
- [readme] Default spectrum filtering thresholds: "Default values are minimum 5 peaks and 250 m/z range. Default values are 101 m/z and 500 m/z, respectively."
- [readme] Spectrum preprocessing rationale and adjustment guidance: "The default settings are intended for clustering bottom-up proteomics data. When analyzing metabolomics or top-down data, these settings likely need to be adjusted accordingly."
- [readme] Intensity scaling options: "Scale the peak intensities by their square root, logarithm, rank, or no scaling. Default is no scaling, with square root scaling often giving good results as well."
- [readme] Role in broader clustering pipeline: "the spectrum vectors are used to construct nearest neighbor indexes for fast similarity searching"
1---2name: high-resolution-mass-spectrometry-data-preprocessing3description: Use when you have acquired high-resolution MS/MS spectra in mzML, mzXML, or MGF format and need to prepare them for large-scale clustering or similarity searching.4license: CC-BY-4.05---67# high-resolution-mass-spectrometry-data-preprocessing89## Summary1011Transform high-resolution tandem mass spectrometry (MS/MS) spectra from mzML, mzXML, or MGF format into low-dimensional hashed feature vectors suitable for fast nearest-neighbor indexing and clustering. This preprocessing step is essential for scaling spectrum similarity searching to millions of spectra without exhaustive pairwise comparisons.1213## When to use1415You have acquired high-resolution MS/MS spectra in mzML, mzXML, or MGF format and need to prepare them for large-scale clustering or similarity searching. Apply this skill when your analysis goal requires finding similar spectra across thousands or millions of spectra efficiently, or when you intend to construct nearest-neighbor indexes to avoid computing all pairwise distances exhaustively.1617## When NOT to use1819- Input spectra are already in low-dimensional vector or feature table form (e.g., quantification matrices from label-free or labeled proteomics).20- Your analysis requires interpretation of individual fragment ion assignments; feature hashing obscures which specific m/z bins contributed to each hash bucket.21- You need to preserve exact mass precision for substructure annotation or metabolite identification; binning and hashing approximate similarity at the cost of exact mass fidelity.2223## Inputs2425- mzML peak files26- mzXML peak files27- MGF peak files2829## Outputs3031- Low-dimensional hashed feature matrix (NumPy array or SciPy sparse matrix format)32- CSV file with spectrum identifiers and corresponding feature vectors33- Filtered spectrum metadata (precursor m/z, charge state, retention time)3435## How to apply3637Load peak files in mzML, mzXML, or MGF format using spectrum-utils or equivalent tools. Apply spectrum-level filtering (e.g., discard spectra with fewer than 5 peaks or spanning less than 250 m/z range; default min_mz 101, max_mz 500) to remove low-quality spectra before vectorization. Bin high-resolution spectra into small mass bins to create sparse, high-dimensional vectors that tightly capture fragment mass values. Apply feature hashing using a non-cryptographic hash function (e.g., MurmurHash3) to map the sparse bins into a fixed, low-dimensional vector space (controlled by the `low_dim` parameter). Optionally scale peak intensities (square root, logarithm, or rank scaling) to improve downstream cosine similarity estimates. Output the resulting hashed feature matrix as a dense or sparse numerical format (NumPy array, SciPy sparse matrix, or CSV). The hashing step preserves cosine similarity between the original high-resolution spectra while reducing dimensionality and memory footprint, making the vectors suitable for fast nearest-neighbor searching.3839## Related tools4041- **falcon** (Full spectrum clustering pipeline that consumes the hashed feature vectors produced by this preprocessing step to perform density-based clustering via nearest-neighbor indexes) — https://github.com/bittremieux/falcon42- **spectrum-utils** (Utility library for reading, filtering, and normalizing MS/MS spectra from mzML, mzXML, and MGF formats prior to feature hashing)4344## Examples4546```47falcon peak/*.mzml falcon --export_representatives --precursor_tol 20 ppm --fragment_tol 0.05 --eps 0.1048```4950## Evaluation signals5152- Output feature vectors have fixed dimensionality equal to the `low_dim` parameter (e.g., all rows in NumPy array or sparse matrix have same number of columns).53- Cosine similarity between hashed vectors approximates the true cosine similarity of the original high-resolution spectra (verify on a representative sample by comparing hashed vs. unhashed cosine scores).54- Memory footprint of the feature matrix is substantially smaller than the original mzML/mzXML files (sparse matrix representation should be ≤ 10% of raw file size for typical proteomics data).55- Filtered spectra retain expected precursor m/z and charge state distributions; no unexpected loss of spectra outside the specified min_mz and max_mz bounds.56- Downstream nearest-neighbor indexing succeeds without out-of-memory errors, and clustering results show expected cluster purity (spectra in the same cluster correspond to the same peptide or compound).5758## Limitations5960- Feature hashing reduces interpretability: the contribution of specific fragment ions to similarity scores is obscured by the hash function, making post-hoc validation of cluster assignments difficult.61- The choice of `low_dim` parameter trades off memory and speed against accuracy; smaller vectors may miss true neighbors in high-dimensional space, resulting in spurious cluster fragmentation.62- Spectrum preprocessing settings (min_peaks, min_mz_range, min_mz, max_mz, scaling method) are optimized for bottom-up proteomics by default; metabolomics or top-down data require manual adjustment of these thresholds.63- Feature hashing is sensitive to the choice of hash function and collision rate; collision-induced artifacts can accumulate if the vector dimensionality is too low relative to the number of mass bins.64- The method assumes mass spectral cosine similarity is the appropriate metric for downstream clustering; other similarity measures (e.g., spectral angle, entropy-based) are not accounted for in the hashing scheme.6566## Evidence6768- [readme] Spectrum vectorization and hashing mechanism: "First, high-resolution spectra are binned and converted to low-dimensional vectors using feature hashing."69- [readme] Input file formats accepted: "falcon takes peak files (in the mzML, mzXML, or MGF format) as input"70- [readme] Feature hashing algorithm details: "the sparse, high-dimensional, vectors are hashed to lower-dimensional vectors by using a hash function (the non-cryptographic MurmurHash3 algorithm) to map the mass bins separately to a small number"71- [readme] Cosine similarity preservation: "This feature hashing conserves the cosine similarity between hashed vectors and can be used to approximate the similarity between the original spectra."72- [readme] Default spectrum filtering thresholds: "Default values are minimum 5 peaks and 250 m/z range. Default values are 101 m/z and 500 m/z, respectively."73- [readme] Spectrum preprocessing rationale and adjustment guidance: "The default settings are intended for clustering bottom-up proteomics data. When analyzing metabolomics or top-down data, these settings likely need to be adjusted accordingly."74- [readme] Intensity scaling options: "Scale the peak intensities by their square root, logarithm, rank, or no scaling. Default is no scaling, with square root scaling often giving good results as well."75- [readme] Role in broader clustering pipeline: "the spectrum vectors are used to construct nearest neighbor indexes for fast similarity searching"
