Spectrum Metadata Extraction

spectrum-metadata-extraction

Summary

Extract and structure spectrum-level metadata (descriptions, scans, precursors, selected ions) from mass spectrometry data into a Parquet-backed tabular format using the mzPeak specification. This skill enables standardized metadata capture and interoperability across different MS instrument vendors and data pipelines.

When to use

When you have raw mass spectrometry data from diverse instrument vendors (Thermo, Sciex, etc.) and need to harmonize and standardize spectrum-level metadata—including scan information, precursor m/z and charge, and ion selection parameters—into a queryable, vendor-agnostic tabular schema prior to downstream spectral analysis, identification, or cross-platform data sharing.

When NOT to use

• If you need read/write capabilities for metadata modification in Python or R—only the Rust implementation supports writing; Python and R implementations support reading only.
• If your MS data is already in a finalized, vendor-locked binary format and you require lossless round-trip conversion without access to the mzPeak Rust toolchain.
• If spectrum-level metadata is minimal or absent (e.g., centroid-only data with no precursor information), as the packed parallel table structure assumes hierarchical metadata is available.

Inputs

• Raw mass spectrometry data files (mzML, vendor proprietary formats: Thermo .raw, Sciex .wiff, etc.)
• Instrument metadata (vendor, model, configuration parameters)
• Spectrum list with scan numbers and precursor information

Outputs

• spectra_metadata.parquet file (Parquet table within mzPeak archive)
• Structured tabular metadata with columns: spectrum_index, scan_number, precursor_mz, precursor_charge, selected_ion_mz, precursor_isolation_window_lower_mz, precursor_isolation_window_upper_mz, and optional vendor-specific fields
• mzpeak_index.json (archive manifest)

How to apply

Load the raw MS data (mzML, proprietary vendor formats, or other MS archive) using the appropriate language-specific mzPeak reader (Rust for full read/write, Python or R arrow implementation for read-only access). Extract spectrum-level fields including spectrum descriptions, scan indices, precursor m/z, precursor charge, selected ion windows, and any instrument-specific metadata using the packed parallel table structure defined in spectra_metadata.parquet. Map vendor-specific terms to controlled vocabulary terms where feasible (using HUPO-PSI CV recommendations) and serialize the result into Parquet format within the mzPeak archive. Validate that all required columns are present and non-null for critical fields (spectrum index, scan number, precursor m/z) and verify that the row count matches the number of spectra in the source data.

Related tools

• Rust mzPeak library (Primary implementation for reading and writing mzPeak files and converting existing MS formats into mzPeak with full metadata extraction) — https://github.com/HUPO-PSI/mzPeak
• pyarrow (Python backend for reading and parsing Parquet-encoded spectrum metadata from mzPeak archives) — https://arrow.apache.org/docs/python/index.html
• arrow (R package) (R backend for reading and extracting spectrum metadata from mzPeak Parquet tables) — https://arrow.apache.org/docs/r/
• OpenMS (Complementary MS data processing framework for validation and manipulation of spectrum metadata) — https://www.openms.de/

Evaluation signals

• All spectrum records are present in spectra_metadata.parquet with row count matching source data spectrum count
• Critical metadata columns (spectrum_index, scan_number, precursor_mz, precursor_charge) are non-null for MS/MS spectra and properly typed (integers for indices/charges, floats for m/z values)
• Precursor isolation windows (lower_mz, upper_mz) satisfy the invariant: lower_mz < precursor_mz < upper_mz for all records
• Controlled vocabulary terms (e.g., polarity, activation method) match HUPO-PSI CV or are traceable to external ontologies
• mzpeak_index.json correctly lists spectra_metadata.parquet and validates against the mzPeak JSON Schema

Limitations

• The R and Python implementations support read-only access to metadata; modifications require the Rust implementation.
• Metadata extraction fidelity depends on vendor format documentation; proprietary or undocumented vendor fields may be lost during conversion.
• The packed parallel table structure assumes hierarchical metadata is present; sparse or degenerate metadata (e.g., missing precursors for all scans) will result in many null values and reduced storage efficiency.
• mzPeak specification is in work-in-progress status with no stability guarantee at this point; schema changes in future versions may require re-extraction of existing archives.

Evidence

• [readme] Spectrum level metadata and file-level metadata. Includes spectrum descriptions, scans, precursors, and selected ions using packed parallel tables.: "Spectrum level metadata and file-level metadata. Includes spectrum descriptions, scans, precursors, and selected ions using packed parallel tables."
• [readme] The primary work shown here is written in Rust at the repository root, including a library for reading and writing mzPeak files, as well as command line tools for converting existing formats into mzPeak: "including a library for reading and writing mzPeak files, as well as command line tools for converting existing formats into mzPeak"
• [readme] There is a separate Python implementation in python/ which is a complete re-implementation for reading mzPeak files using [pyarrow]: "complete re-implementation for reading mzPeak files using [pyarrow]"
• [readme] There is also an R implementation in R/, which is also a complete re-implementation using the [arrow] for reading only at this time.: "complete re-implementation using the [arrow] for reading only at this time"
• [readme] mzPeak file-level metadata, including descriptions of the file's contents, the instrumentation, software, and data transformation pipeline are stored in the Parquet metadata segment as JSON documents.: "descriptions of the file's contents, the instrumentation, software, and data transformation pipeline are stored in the Parquet metadata segment as JSON documents"
• [readme] NOTE: This is a work in progress, no stability is guaranteed at this point.: "This is a work in progress, no stability is guaranteed at this point."