mass-spectrometry-workflow-orchestration-snakemake
Summary
Orchestrate end-to-end high-dimensional mass spectrometry (MS) data processing pipelines using Snakemake to automatically execute feature detection, alignment, CCS calibration, isotope detection, and MS/MS deconvolution on mzML input files, producing HDF5-formatted output with characterized features, spectra, and isotopic signatures across study samples.
When to use
When you have a collection of mzML.gz files from a multidimensional MS instrument (e.g., ion mobility–time-of-flight MS) and need to apply a complete, reproducible workflow that detects features across N-dimensional data space, aligns them across samples, calibrates collision cross sections, and deconvolves tandem mass spectra without manually invoking individual processing steps.
When NOT to use
- Input data is already a feature table or has been pre-aligned; use this skill only on raw mzML.gz files to avoid redundant re-processing.
- Single-dimensional MS data (e.g., 2D LC–MS without ion mobility) where multi-dimensional feature separation offers minimal benefit.
- Real-time or streaming MS acquisition where Snakemake batch orchestration is inappropriate; use the deimos Python API directly instead.
Inputs
- mzML.gz files (N-dimensional MS data with accession mappings for drift_time and retention_time)
- config.yaml (Snakemake workflow configuration specifying dataset parameters, thresholds, and output paths)
- CCS calibration reference dataset (tuning file with known CCS values, e.g., example_tune_pos.h5)
Outputs
- HDF5 feature table (features dataset with detected, aligned features characterized by m/z, drift_time, retention_time, and intensity)
- HDF5 MS1 dataset (thresholded, detected peaks)
- HDF5 MS2 dataset (deconvolved tandem mass spectra by feature)
- HDF5 isotopes dataset (isotopic signatures with parent–daughter relationships and abundance ratios)
- CCS calibration model (fit parameters and r² metric for drift-time-to-CCS conversion)
How to apply
Clone the DEIMoS repository and activate a conda environment with required dependencies. Prepare a config.yaml file specifying dataset-specific parameters (e.g., thresholds, calibration standards). Place mzML.gz input files in the input/ directory. Invoke the DEIMoS CLI with the configuration file and number of processor cores (e.g., deimos --config config.yaml --cores N). Snakemake automatically constructs and executes the rule dependency graph, detecting peaks, aligning features across samples using all N-dimensional coordinates (m/z, drift time, retention time), fitting CCS calibrations (typically yielding r² ≥ 0.9999), identifying isotopic signatures with ≥3 members as a quality screen, and deconvolving MS/MS spectra. Monitor rule completion and verify that all expected HDF5 output files in output/ contain non-zero row counts in the ms1, ms2, features, and isotopes datasets.
Related tools
- DEIMoS (Core Python API and CLI providing feature detection, alignment, CCS calibration, isotope detection, and MS/MS deconvolution algorithms) — http://github.com/pnnl/deimos
- Snakemake (Workflow orchestration engine that parses config.yaml, manages rule dependencies, parallelizes execution across cores, and handles input/output staging)
- conda (Virtual environment and dependency management system used to install DEIMoS and all transitive Python/system dependencies)
- ProteoWizard msconvert (Format conversion utility to convert raw instrument files (e.g., .raw, .d) to mzML.gz before workflow input)
Examples
deimos --config config.yaml --cores 8
Evaluation signals
- Snakemake rule DAG completes without errors; all expected rules (peak_detection, feature_alignment, ccs_calibration, isotope_detection, ms2_deconvolution) execute to completion.
- All output HDF5 files (ms1, ms2, features, isotopes keys) are present in output/ and contain non-zero row counts; features table has ≥1 row per sample.
- CCS calibration r² metric is ≥0.9999, confirming high-quality drift-time-to-CCS model fit.
- Isotopic signatures in the isotopes HDF5 dataset have ≥3 members and consistent mass defect relationships (e.g., expected ¹³C, ¹⁵N shifts).
- Feature retention times and m/z values fall within expected ranges for the instrument and sample; no NaN or inf values in aligned feature table.
Limitations
- DEIMoS assumes mzML.gz files carry explicit cvParam accession identifiers (e.g., MS:1000016 for retention_time, MS:1002476 for drift_time); missing or non-standard accessions will cause workflow failure.
- CCS calibration requires a reference tuning file with known m/z and CCS standards; absence or poor-quality calibration standards degrades alignment confidence.
- Algorithm performance is dependent on tuning of threshold parameters (e.g., intensity thresholds of 100–1000 for peak detection); no universal threshold works across all instrument types or sample matrices.
- No changelog provided in repository, limiting visibility into versioning and backward compatibility across releases.
Evidence
- [other] Snakemake workflow execution: "Invoke the DEIMoS CLI with the configuration file, specifying the number of cores and execution mode (local or cluster) as needed: deimos --config config.yaml --cores N."
- [intro] Multi-dimensional feature separation and alignment: "algorithm implementations simultaneously utilize all dimensions to (i) offer greater separation between features, thus improving detection sensitivity"
- [other] Output HDF5 format and datasets: "Verify successful completion by confirming all output HDF5 files are present in output/ and contain expected datasets (ms1, ms2, features, isotopes) with non-zero row counts."
- [intro] Complete workflow rule chain: "Functionality includes feature detection, feature alignment, collision cross section (CCS) calibration, isotope detection, and MS/MS spectral deconvolution"
- [results] Input data format and accession mapping: "data = deimos.load('example_data.mzML.gz', accession={'retention_time': 'MS:1000016', 'drift_time': 'MS:1002476'})"
- [results] Isotope filtering criterion: "A good first screening is to only consider those isotopic signatures with at least 3 members."
- [intro] N-dimensional agnostic processing: "DEIMoS operates on N-dimensional data, largely agnostic to acquisition instrumentation"
1---2name: mass-spectrometry-workflow-orchestration-snakemake-23description: Use when when you have a collection of mzML.gz files from a multidimensional MS instrument (e.4license: CC-BY-4.05---67# mass-spectrometry-workflow-orchestration-snakemake89## Summary1011Orchestrate end-to-end high-dimensional mass spectrometry (MS) data processing pipelines using Snakemake to automatically execute feature detection, alignment, CCS calibration, isotope detection, and MS/MS deconvolution on mzML input files, producing HDF5-formatted output with characterized features, spectra, and isotopic signatures across study samples.1213## When to use1415When you have a collection of mzML.gz files from a multidimensional MS instrument (e.g., ion mobility–time-of-flight MS) and need to apply a complete, reproducible workflow that detects features across N-dimensional data space, aligns them across samples, calibrates collision cross sections, and deconvolves tandem mass spectra without manually invoking individual processing steps.1617## When NOT to use1819- Input data is already a feature table or has been pre-aligned; use this skill only on raw mzML.gz files to avoid redundant re-processing.20- Single-dimensional MS data (e.g., 2D LC–MS without ion mobility) where multi-dimensional feature separation offers minimal benefit.21- Real-time or streaming MS acquisition where Snakemake batch orchestration is inappropriate; use the deimos Python API directly instead.2223## Inputs2425- mzML.gz files (N-dimensional MS data with accession mappings for drift_time and retention_time)26- config.yaml (Snakemake workflow configuration specifying dataset parameters, thresholds, and output paths)27- CCS calibration reference dataset (tuning file with known CCS values, e.g., example_tune_pos.h5)2829## Outputs3031- HDF5 feature table (features dataset with detected, aligned features characterized by m/z, drift_time, retention_time, and intensity)32- HDF5 MS1 dataset (thresholded, detected peaks)33- HDF5 MS2 dataset (deconvolved tandem mass spectra by feature)34- HDF5 isotopes dataset (isotopic signatures with parent–daughter relationships and abundance ratios)35- CCS calibration model (fit parameters and r² metric for drift-time-to-CCS conversion)3637## How to apply3839Clone the DEIMoS repository and activate a conda environment with required dependencies. Prepare a config.yaml file specifying dataset-specific parameters (e.g., thresholds, calibration standards). Place mzML.gz input files in the input/ directory. Invoke the DEIMoS CLI with the configuration file and number of processor cores (e.g., `deimos --config config.yaml --cores N`). Snakemake automatically constructs and executes the rule dependency graph, detecting peaks, aligning features across samples using all N-dimensional coordinates (m/z, drift time, retention time), fitting CCS calibrations (typically yielding r² ≥ 0.9999), identifying isotopic signatures with ≥3 members as a quality screen, and deconvolving MS/MS spectra. Monitor rule completion and verify that all expected HDF5 output files in output/ contain non-zero row counts in the ms1, ms2, features, and isotopes datasets.4041## Related tools4243- **DEIMoS** (Core Python API and CLI providing feature detection, alignment, CCS calibration, isotope detection, and MS/MS deconvolution algorithms) — http://github.com/pnnl/deimos44- **Snakemake** (Workflow orchestration engine that parses config.yaml, manages rule dependencies, parallelizes execution across cores, and handles input/output staging)45- **conda** (Virtual environment and dependency management system used to install DEIMoS and all transitive Python/system dependencies)46- **ProteoWizard msconvert** (Format conversion utility to convert raw instrument files (e.g., .raw, .d) to mzML.gz before workflow input)4748## Examples4950```51deimos --config config.yaml --cores 852```5354## Evaluation signals5556- Snakemake rule DAG completes without errors; all expected rules (peak_detection, feature_alignment, ccs_calibration, isotope_detection, ms2_deconvolution) execute to completion.57- All output HDF5 files (ms1, ms2, features, isotopes keys) are present in output/ and contain non-zero row counts; features table has ≥1 row per sample.58- CCS calibration r² metric is ≥0.9999, confirming high-quality drift-time-to-CCS model fit.59- Isotopic signatures in the isotopes HDF5 dataset have ≥3 members and consistent mass defect relationships (e.g., expected ¹³C, ¹⁵N shifts).60- Feature retention times and m/z values fall within expected ranges for the instrument and sample; no NaN or inf values in aligned feature table.6162## Limitations6364- DEIMoS assumes mzML.gz files carry explicit cvParam accession identifiers (e.g., MS:1000016 for retention_time, MS:1002476 for drift_time); missing or non-standard accessions will cause workflow failure.65- CCS calibration requires a reference tuning file with known m/z and CCS standards; absence or poor-quality calibration standards degrades alignment confidence.66- Algorithm performance is dependent on tuning of threshold parameters (e.g., intensity thresholds of 100–1000 for peak detection); no universal threshold works across all instrument types or sample matrices.67- No changelog provided in repository, limiting visibility into versioning and backward compatibility across releases.6869## Evidence7071- [other] Snakemake workflow execution: "Invoke the DEIMoS CLI with the configuration file, specifying the number of cores and execution mode (local or cluster) as needed: deimos --config config.yaml --cores N."72- [intro] Multi-dimensional feature separation and alignment: "algorithm implementations simultaneously utilize all dimensions to (i) offer greater separation between features, thus improving detection sensitivity"73- [other] Output HDF5 format and datasets: "Verify successful completion by confirming all output HDF5 files are present in output/ and contain expected datasets (ms1, ms2, features, isotopes) with non-zero row counts."74- [intro] Complete workflow rule chain: "Functionality includes feature detection, feature alignment, collision cross section (CCS) calibration, isotope detection, and MS/MS spectral deconvolution"75- [results] Input data format and accession mapping: "data = deimos.load('example_data.mzML.gz', accession={'retention_time': 'MS:1000016', 'drift_time': 'MS:1002476'})"76- [results] Isotope filtering criterion: "A good first screening is to only consider those isotopic signatures with at least 3 members."77- [intro] N-dimensional agnostic processing: "DEIMoS operates on N-dimensional data, largely agnostic to acquisition instrumentation"
HolobiomicsLab 