generic-hdf5-library-api-usage
Summary
Access and traverse multidimensional mass spectrometry data stored in HDF5 format using language-agnostic libraries (h5py for Python, rhdf5 for R) to extract metadata, structural information, and spectral arrays without proprietary vendor tools. This enables cross-language, cross-platform reproducibility of MS data workflows.
When to use
When you have converted multidimensional MS data to MZA HDF5 format and need to programmatically read spectra, metadata, and ion mobility or mass-to-charge arrays in Python or R without vendor-specific dependencies. Also applicable when you need to validate the internal structure of an HDF5 MS file or compare outputs across programming languages.
When NOT to use
- Input is raw proprietary vendor data (.raw, .d, or .mzML); use MZA conversion tool first, not generic HDF5 libraries directly.
- You require real-time streaming or on-disk editing of spectra; HDF5 generic libraries read data into memory and are optimized for batch access.
- You need vendor-specific instrument metadata (e.g., calibration parameters, instrument model details) not stored in the MZA HDF5 structure; consult original vendor files.
Inputs
- MZA-converted HDF5 file (.mza extension) containing metadata dataset and spectral arrays
- Optionally: intensity threshold value (numeric, default behavior includes all signals)
- Optionally: scan number filter or ion mobility/retention time range
Outputs
- Extracted metadata table (DataFrame or list of dictionaries with Scan, MSLevel, RetentionTime, IonMobilityTime, etc.)
- Spectral arrays (m/z and intensity pairs per spectrum, or mzbin indices linked to Full_mz_array)
- File structure report (JSON or text) documenting group hierarchy, dataset names, shapes, and data types
- Optionally: serialized subsets or filtered spectra in HDF5, CSV, or JSON format
How to apply
Load the MZA HDF5 file using h5py (Python) or rhdf5 (R) and traverse the file hierarchy to enumerate top-level groups (Arrays_intensity, Arrays_mz, Arrays_mzbin, and metadata table). Extract dimensions, data types, and dataset shapes for each spectrum. For ion mobility data, resolve mzbin indices against the Full_mz_array dataset. Apply intensity thresholds (if desired) by filtering spectra where TIC or individual peak intensities fall below a user-defined minimum. Serialize extracted structure and arrays to a human-readable format (JSON, CSV, or HDF5 subset) for downstream analysis or cross-validation. The rationale is that HDF5's generic libraries provide vendor-independent access while preserving the original data structure and enabling reproducible workflows across platforms.
Related tools
- h5py (Generic HDF5 file read/write API for Python; loads MZA files and exposes metadata, intensity, and m/z datasets as NumPy arrays)
- rhdf5 (Generic HDF5 file read/write API for R; loads MZA files and exposes spectral arrays and metadata as R data structures)
- MZA (Prerequisite converter tool that transforms multidimensional MS data from vendor formats (.d, .raw, .mzML) into the simple HDF5-based MZA structure) — https://github.com/PNNL-m-q/mza
- hdf5plugin (Python extension enabling transparent decompression of HDF5 compressed datasets in MZA files)
- NumPy (Python array manipulation library; provides in-memory representation of spectral arrays returned by h5py)
Examples
import h5py
import numpy as np
with h5py.File('data.mza', 'r') as f:
metadata = f['Metadata'][:]
scan_630_intensity = f['Arrays_intensity/630'][:]
scan_630_mz = f['Arrays_mz/630'][:]
Evaluation signals
- Successfully enumerated all top-level groups (Metadata, Arrays_intensity, Arrays_mz, Arrays_mzbin if present, Full_mz_array if present) with correct dataset/group counts matching the number of spectra in the file.
- Metadata table row count equals number of spectra; all required columns (Scan, MSLevel, RetentionTime, IonMobilityBin, IonMobilityTime if IM data) are present and non-null where expected.
- For each spectrum, intensity and m/z arrays have matching lengths; intensity values are numeric and positive; m/z values are in expected range (typically 50–2000 m/z).
- Cross-language consistency: identical metadata and spectral arrays retrieved from the same MZA file using h5py and rhdf5 (byte-level equality or within floating-point tolerance).
- For ion mobility data, mzbin indices successfully resolve to Full_mz_array without out-of-bounds errors; TIC reconstructed from intensity arrays matches metadata TIC column within <1% relative error.
Limitations
- MZA HDF5 files omit zero-intensity values to save storage; reconstructing full spectra with zeros requires knowledge of the instrument's m/z binning scheme, which is not stored.
- For files with many spectra (>100k), data may be partitioned across multiple HDF5 groups (e.g., Arrays_intensity1, Arrays_intensity2); the MzaPath column in metadata identifies the correct group partition for each spectrum.
- CCS calibration coefficients (CCScalDT, CCScalSLIM, CCScalTIMS) are included only if detected during MZA conversion; their presence is not guaranteed, and ion mobility interpretation is incomplete without them.
- Generic HDF5 libraries read entire datasets into memory; very large MZA files (>10 GB) may exceed available RAM on resource-constrained systems.
- The MZA file structure does not preserve raw detector counts or instrument-specific quality metrics; only the converted simple structure (metadata + arrays) is accessible via HDF5 libraries.
Evidence
- [readme] Once converted, MZA files can be easily accessed from any programming language and operating system using generic HDF5 libraries available (e.g., h5py and rhdf5).: "Once converted, MZA files can be easily accessed from any programming language and operating system using generic HDF5 libraries available (e.g., h5py and rhdf5)."
- [readme] The MZA file structure is simple: a metadata table and two groups with 1D arrays stored individually as HDF5 datasets per spectrum.: "The MZA file structure is simple: a metadata table and two groups with 1D arrays stored individually as HDF5 datasets per spectrum."
- [readme] Metadata (HDF5 dataset): each row in the metadata table represents a spectrum and the columns represent the properties of the spectrum such as scan number, MS level, activation, retention time, ion mobility arrival time, etc.: "Metadata (HDF5 dataset): each row in the metadata table represents a spectrum and the columns represent the properties of the spectrum such as scan number, MS level, activation, retention time, ion"
- [readme] Spectra are stored omitting zero-intensity values and as two jagged arrays, one in each corresponding group and named as the "Scan" value in the metadata table: Arrays_intensity (HDF5 group) contains 1D arrays with intensity values. Arrays_mz (HDF5 group) contains 1D arrays with mass-to-charge (m/z) values.: "Spectra are stored omitting zero-intensity values and as two jagged arrays, one in each corresponding group and named as the "Scan" value in the metadata table: Arrays_intensity (HDF5 group) contains"
- [readme] For IM spectra the m/z dimension is stored as indexes (mzbins): Arrays_mzbin (HDF5 group) contains 1D arrays with indexes to Full_mz_array.: "For IM spectra the m/z dimension is stored as indexes (mzbins): Arrays_mzbin (HDF5 group) contains 1D arrays with indexes to Full_mz_array."
- [readme] Example scripts are provided in the respective folders. R requires the rhdf5 package and Python requires the h5py, hdf5plugin, numpy and matplotlib packages.: "Example scripts are provided in the respective folders. R requires the rhdf5 package and Python requires the h5py, hdf5plugin, numpy and matplotlib packages."
- [readme] Partitions may be created for files with too many spectra. Having too many datasets within a single group can slow down data reading performance. HDF5 is more efficient when using multiple groups instead of storing many datasets within one group.: "Partitions may be created for files with too many spectra. Having too many datasets within a single group can slow down data reading performance. HDF5 is more efficient when using multiple groups"
1---2name: generic-hdf5-library-api-usage3description: Use when when you have converted multidimensional MS data to MZA HDF5 format and need to programmatically read spectra, metadata, and ion mobility or mass-to-charge arrays in Python or R without vendor-specific dependencies.4license: CC-BY-4.05---67# generic-hdf5-library-api-usage89## Summary1011Access and traverse multidimensional mass spectrometry data stored in HDF5 format using language-agnostic libraries (h5py for Python, rhdf5 for R) to extract metadata, structural information, and spectral arrays without proprietary vendor tools. This enables cross-language, cross-platform reproducibility of MS data workflows.1213## When to use1415When you have converted multidimensional MS data to MZA HDF5 format and need to programmatically read spectra, metadata, and ion mobility or mass-to-charge arrays in Python or R without vendor-specific dependencies. Also applicable when you need to validate the internal structure of an HDF5 MS file or compare outputs across programming languages.1617## When NOT to use1819- Input is raw proprietary vendor data (.raw, .d, or .mzML); use MZA conversion tool first, not generic HDF5 libraries directly.20- You require real-time streaming or on-disk editing of spectra; HDF5 generic libraries read data into memory and are optimized for batch access.21- You need vendor-specific instrument metadata (e.g., calibration parameters, instrument model details) not stored in the MZA HDF5 structure; consult original vendor files.2223## Inputs2425- MZA-converted HDF5 file (.mza extension) containing metadata dataset and spectral arrays26- Optionally: intensity threshold value (numeric, default behavior includes all signals)27- Optionally: scan number filter or ion mobility/retention time range2829## Outputs3031- Extracted metadata table (DataFrame or list of dictionaries with Scan, MSLevel, RetentionTime, IonMobilityTime, etc.)32- Spectral arrays (m/z and intensity pairs per spectrum, or mzbin indices linked to Full_mz_array)33- File structure report (JSON or text) documenting group hierarchy, dataset names, shapes, and data types34- Optionally: serialized subsets or filtered spectra in HDF5, CSV, or JSON format3536## How to apply3738Load the MZA HDF5 file using h5py (Python) or rhdf5 (R) and traverse the file hierarchy to enumerate top-level groups (Arrays_intensity, Arrays_mz, Arrays_mzbin, and metadata table). Extract dimensions, data types, and dataset shapes for each spectrum. For ion mobility data, resolve mzbin indices against the Full_mz_array dataset. Apply intensity thresholds (if desired) by filtering spectra where TIC or individual peak intensities fall below a user-defined minimum. Serialize extracted structure and arrays to a human-readable format (JSON, CSV, or HDF5 subset) for downstream analysis or cross-validation. The rationale is that HDF5's generic libraries provide vendor-independent access while preserving the original data structure and enabling reproducible workflows across platforms.3940## Related tools4142- **h5py** (Generic HDF5 file read/write API for Python; loads MZA files and exposes metadata, intensity, and m/z datasets as NumPy arrays)43- **rhdf5** (Generic HDF5 file read/write API for R; loads MZA files and exposes spectral arrays and metadata as R data structures)44- **MZA** (Prerequisite converter tool that transforms multidimensional MS data from vendor formats (.d, .raw, .mzML) into the simple HDF5-based MZA structure) — https://github.com/PNNL-m-q/mza45- **hdf5plugin** (Python extension enabling transparent decompression of HDF5 compressed datasets in MZA files)46- **NumPy** (Python array manipulation library; provides in-memory representation of spectral arrays returned by h5py)4748## Examples4950```51import h5py52import numpy as np53with h5py.File('data.mza', 'r') as f:54 metadata = f['Metadata'][:]55 scan_630_intensity = f['Arrays_intensity/630'][:]56 scan_630_mz = f['Arrays_mz/630'][:]57```5859## Evaluation signals6061- Successfully enumerated all top-level groups (Metadata, Arrays_intensity, Arrays_mz, Arrays_mzbin if present, Full_mz_array if present) with correct dataset/group counts matching the number of spectra in the file.62- Metadata table row count equals number of spectra; all required columns (Scan, MSLevel, RetentionTime, IonMobilityBin, IonMobilityTime if IM data) are present and non-null where expected.63- For each spectrum, intensity and m/z arrays have matching lengths; intensity values are numeric and positive; m/z values are in expected range (typically 50–2000 m/z).64- Cross-language consistency: identical metadata and spectral arrays retrieved from the same MZA file using h5py and rhdf5 (byte-level equality or within floating-point tolerance).65- For ion mobility data, mzbin indices successfully resolve to Full_mz_array without out-of-bounds errors; TIC reconstructed from intensity arrays matches metadata TIC column within <1% relative error.6667## Limitations6869- MZA HDF5 files omit zero-intensity values to save storage; reconstructing full spectra with zeros requires knowledge of the instrument's m/z binning scheme, which is not stored.70- For files with many spectra (>100k), data may be partitioned across multiple HDF5 groups (e.g., Arrays_intensity1, Arrays_intensity2); the MzaPath column in metadata identifies the correct group partition for each spectrum.71- CCS calibration coefficients (CCScalDT, CCScalSLIM, CCScalTIMS) are included only if detected during MZA conversion; their presence is not guaranteed, and ion mobility interpretation is incomplete without them.72- Generic HDF5 libraries read entire datasets into memory; very large MZA files (>10 GB) may exceed available RAM on resource-constrained systems.73- The MZA file structure does not preserve raw detector counts or instrument-specific quality metrics; only the converted simple structure (metadata + arrays) is accessible via HDF5 libraries.7475## Evidence7677- [readme] Once converted, MZA files can be easily accessed from any programming language and operating system using generic HDF5 libraries available (e.g., h5py and rhdf5).: "Once converted, MZA files can be easily accessed from any programming language and operating system using generic HDF5 libraries available (e.g., h5py and rhdf5)."78- [readme] The MZA file structure is simple: a metadata table and two groups with 1D arrays stored individually as HDF5 datasets per spectrum.: "The MZA file structure is simple: a metadata table and two groups with 1D arrays stored individually as HDF5 datasets per spectrum."79- [readme] Metadata (HDF5 dataset): each row in the metadata table represents a spectrum and the columns represent the properties of the spectrum such as scan number, MS level, activation, retention time, ion mobility arrival time, etc.: "Metadata (HDF5 dataset): each row in the metadata table represents a spectrum and the columns represent the properties of the spectrum such as scan number, MS level, activation, retention time, ion"80- [readme] Spectra are stored omitting zero-intensity values and as two jagged arrays, one in each corresponding group and named as the "Scan" value in the metadata table: Arrays_intensity (HDF5 group) contains 1D arrays with intensity values. Arrays_mz (HDF5 group) contains 1D arrays with mass-to-charge (m/z) values.: "Spectra are stored omitting zero-intensity values and as two jagged arrays, one in each corresponding group and named as the "Scan" value in the metadata table: Arrays_intensity (HDF5 group) contains"81- [readme] For IM spectra the m/z dimension is stored as indexes (mzbins): Arrays_mzbin (HDF5 group) contains 1D arrays with indexes to Full_mz_array.: "For IM spectra the m/z dimension is stored as indexes (mzbins): Arrays_mzbin (HDF5 group) contains 1D arrays with indexes to Full_mz_array."82- [readme] Example scripts are provided in the respective folders. R requires the rhdf5 package and Python requires the h5py, hdf5plugin, numpy and matplotlib packages.: "Example scripts are provided in the respective folders. R requires the rhdf5 package and Python requires the h5py, hdf5plugin, numpy and matplotlib packages."83- [readme] Partitions may be created for files with too many spectra. Having too many datasets within a single group can slow down data reading performance. HDF5 is more efficient when using multiple groups instead of storing many datasets within one group.: "Partitions may be created for files with too many spectra. Having too many datasets within a single group can slow down data reading performance. HDF5 is more efficient when using multiple groups"
HolobiomicsLab 