Python Binary Io And Serialization

Python Binary I/O and Serialization

Summary

Parse and reconstruct mass spectrometry spectral data from binary or columnar file formats using PyArrow and the PyData stack, enabling read-only access to mzPeak files and conversion of raw spectra into Python-native data structures. This skill addresses the need to deserialize compact, archive-based MS data formats into analysis-ready NumPy arrays and pandas DataFrames.

When to use

When you have mzPeak files (Parquet-based archives in uncompressed ZIP containers) or other PyArrow-compatible columnar formats containing mass spectrometry spectra, and you need to extract and decode spectral data arrays (m/z values, intensities) into Python memory for downstream analysis, filtering, or visualization. Use this when read-only access is acceptable and you want to avoid implementing format-specific binary parsing logic.

When NOT to use

• Writing or modifying mzPeak files—the Python implementation supports reading only; use the Rust implementation for write operations.
• Working with already-decoded spectral data or feature tables in memory—this skill is for initial deserialization from archive format, not downstream processing.
• Formats other than mzPeak or Arrow-compatible columnar storage—use format-specific parsers for mzML, mzXML, or netCDF.
• Real-time or streaming applications requiring random access to individual spectra without full archive decompression—mzPeak supports point layout for this, but architecture must account for ZIP constraints.

Inputs

• mzPeak archive (ZIP file containing Parquet tables and mzpeak_index.json)
• Parquet files (spectra_metadata.parquet, spectra_data.parquet, chromatograms_metadata.parquet, chromatograms_data.parquet)
• mzPeak file format specification (HUPO-PSI reference or inline schema documentation)

Outputs

• Decoded spectral data as NumPy arrays (m/z and intensity parallel arrays)
• Spectrum metadata as pandas DataFrames (spectrum descriptions, scan indices, precursor info)
• Reconstructed m/z spacing models for null-marked regions (polynomial coefficients or local medians)
• Python-native spectrum objects or tabular representations ready for analysis

How to apply

Set up a Python environment with pyarrow and PyData stack dependencies (NumPy, pandas). Open the mzPeak archive and locate the relevant Parquet files (spectra_metadata.parquet, spectra_data.parquet, or spectra_peaks.parquet) using the mzpeak_index.json manifest. Use PyArrow's Table and RecordBatch readers to deserialize columnar data, then decode spectral arrays by handling packed parallel table schemas, zero run stripping, null marking, and point/chunked layouts as appropriate to your data. Validate reconstruction by comparing decoded m/z spacing and intensity distributions against reference spectra or expected signal properties (peak apex positions, centroid calculations should be unaffected by lossless transformations).

Related tools

• PyArrow (Core library for reading and deserializing Parquet columnar data from mzPeak archives; handles null marking, array decoding, and schema validation) — https://arrow.apache.org/docs/python/index.html
• NumPy (Host and manipulate decoded m/z and intensity arrays after PyArrow deserialization; supports mathematical operations and peak detection on spectral data)
• pandas (Convert Parquet metadata tables to DataFrames for spectrum-level annotations, filtering by scan properties, and joining spectral data with metadata)
• mzPeak Rust implementation (Reference implementation providing conversion tools and write support for creating mzPeak files; Python implementation is a read-only re-implementation based on this) — https://github.com/HUPO-PSI/mzPeak

Examples

import pyarrow.parquet as pq; import zipfile; z = zipfile.ZipFile('sample.mzpeak'); metadata = pq.read_table(z.open('spectra_metadata.parquet')); data = pq.read_table(z.open('spectra_data.parquet')); df = metadata.to_pandas(); spectra = {row['spectrum_id']: data.filter(data['spectrum_index'] == row['spectrum_index']).to_pandas() for _, row in df.iterrows()}

Evaluation signals

• Decoded m/z and intensity arrays match reference test spectra in shape, data type, and value ranges (peak positions, centroid calculations invariant under lossless transformations)
• Null-marked regions reconstruct correctly: repopulated m/z values fall within tolerance of original or fitted m/z spacing model; intensity values match non-null points
• Zero run stripping validation: reconstructed spectra contain original non-zero intensity points and flanking zeros, with no missing peaks or artifacts
• Parquet schema validation: spectrum_index, mz, intensity columns successfully read; packed parallel table structures (scan, precursor, selectedIon groups) parse without null type errors
• Metadata-data joins succeed: spectrum count in metadata matches row count in spectra_data; scan indices and precursor references are consistent across files

Limitations

• Read-only access: The Python implementation does not support writing mzPeak files; writing operations require the Rust implementation.
• Work-in-progress format: mzPeak specification is still in draft status; no stability is guaranteed, and breaking changes to the Parquet schema or ZIP structure may require code updates.
• Zero run stripping and null marking introduce lossy approximations: m/z spacing is reconstructed using a polynomial model (δmz ~ β₀ + β₁·mz + β₂·mz²), which may not hold for all instruments or m/z ranges; edge cases with only 1–2 non-zero points cannot be meaningfully modeled.
• Large file handling: ZIP-based archives do not support true streaming; entire file must be decompressed into memory or temporary storage before Parquet reading.
• Chunked layout complexity: Numpress-compressed data (available in chunked layout) requires additional decompression logic beyond standard Parquet column reading; not all compressions may be supported in the Python implementation.

Evidence

• [intro] The Python implementation is a complete re-implementation for reading mzPeak files using pyarrow and the PyData stack, with read-only functionality.: "complete re-implementation for reading mzPeak files using pyarrow, and the PyData stack. The Python codebase does not support writing at this"
• [readme] mzPeak is a Parquet-based archive format with specific file structure including metadata and spectral data tables.: "mzPeak is a archive of multiple Parquet files, stored directly in an uncompressed ZIP"
• [readme] Spectral data is stored using point or chunked layouts with zero run stripping and optional null marking to compress sparse m/z regions.: "When storing spectrum data, some vendors will produce arrays with lots of "empty" regions filled with zero intensity values... all but the first and last zero intensity points are removed. This is"
• [readme] Null-marked m/z values are reconstructed using a polynomial model fitted to non-null points in order to recover near-lossless representation.: "fit a simple m/z spacing model using weighted least squares of the form: δ mz ∼ β₀ + β₁ mz + β₂ mz² + ϵ"
• [other] The core workflow involves parsing the mzPeak specification and implementing file I/O to return decoded spectra.: "Parse the mzPeak file format specification from the HUPO-PSI repository. Implement file I/O using pyarrow to read mzPeak binary/columnar data structures. Decode spectral data arrays (m/z values,"
• [other] Validation requires comparing reconstructed spectra to reference test files.: "Validate that the implementation correctly reproduces spectra from reference mzPeak test files."