File Format Compliance Validation

file-format-compliance-validation

Summary

Validate that prototype implementations of a mass spectrometry data format produce files conforming to all mandatory fields and structural elements defined in a published specification. This skill ensures interoperability and correctness by systematically comparing generated files against a normative specification document.

When to use

Apply this skill when you have generated or received mzPeak files from a Rust, Python, R, or other implementation and need to verify they comply with the published HUPO-PSI specification before integration into a production workflow, data repository, or downstream analysis pipeline. Use it particularly when validating prototype or work-in-progress implementations that carry no stability guarantee.

When NOT to use

• When validating files generated by a stable, released implementation with known compliance history — use this skill only for prototype, work-in-progress, or newly integrated implementations.
• When the mzPeak specification has not yet been formally published and you require production-grade stability guarantees — the specification is currently a HUPO-PSI working draft with no stability guarantee.
• When you are validating legacy mzML or other non-mzPeak formats — this skill is specific to mzPeak archive structure, Parquet schemas, and zero-run stripping / null-marking logic.

Inputs

• mzPeak file (ZIP archive of Parquet files with mzpeak_index.json, spectra_metadata.parquet, spectra_data.parquet, optional spectra_peaks.parquet, chromatograms_metadata.parquet, chromatograms_data.parquet)
• mzPeak specification document (published draft from https://hupo-psi.github.io/mzPeak-specification/)
• JSON Schema definitions from schema/ directory in specification repository

Outputs

• Compliance report (document or structured output listing mandatory fields, presence status, and overall specification conformance status)
• Field inventory (structured list of detected vs. required fields)
• Structural validation results (confirmation of archive layout, Parquet table schemas, and JSON metadata format)

How to apply

Retrieve the authoritative draft specification document from https://hupo-psi.github.io/mzPeak-specification/ and extract the list of mandatory fields and required structural elements (e.g., mzpeak_index.json, spectra_metadata.parquet, spectra_data.parquet with zero-run stripping and null-marking logic). Obtain a sample mzPeak file generated by the implementation under test. Parse the file using the implementation's native read functions (e.g., Rust library or pyarrow for Python) to extract and inspect its internal structure, field contents, and data layout (point vs. chunked). Cross-check each parsed field and structural element against the specification's mandatory requirements, documenting presence, absence, and correct encoding. Generate a compliance report that lists which mandatory fields are present or absent and compute an overall conformance status. Note that mzPeak is a ZIP archive of Parquet files with file-level metadata stored as JSON in the Parquet metadata segment, so validation must check both the archive structure and the content of each Parquet table.

Related tools

• Rust mzPeak library (Generate and read mzPeak files for validation; provides native read/write API for inspecting archive structure and Parquet table contents) — https://github.com/HUPO-PSI/mzPeak
• pyarrow (Read and parse Parquet tables from mzPeak archives in Python; extract and inspect field contents and schemas) — https://arrow.apache.org/docs/python/index.html
• arrow (R package) (Read and parse Parquet tables from mzPeak archives in R; enable compliance checking in R workflows) — https://arrow.apache.org/docs/r/
• OpenMS (Reference implementation and command-line tools for format conversion and validation)

Evaluation signals

• Presence and correct encoding of all mandatory Parquet files (mzpeak_index.json, spectra_metadata.parquet, spectra_data.parquet, chromatograms_metadata.parquet, chromatograms_data.parquet)
• mzpeak_index.json is valid JSON and references all present Parquet files with correct controlled vocabulary terms
• Each Parquet file conforms to its declared schema (packed parallel tables, proper struct/group nesting, correct data types)
• File-level metadata in Parquet metadata segment is valid JSON and matches JSON Schema definitions in the specification repository
• Zero-run stripping and null-marking logic correctly applied to profile spectrum data: null m/z and intensity pairs match expected reconstruction accuracy with learned quadratic spacing model

Limitations

• The mzPeak specification is a working draft with no stability guarantee; compliance rules may change as the specification evolves.
• Python and R implementations currently support read-only access, so validation must use the Rust implementation or command-line tools for generating test files.
• Validation of null-marking reconstruction accuracy requires fitting a weighted-least-squares m/z spacing model and computing reconstruction error, which may require custom code beyond the libraries themselves.
• The specification is distributed across multiple Markdown files in the docs/ tree; extracting a complete normative ruleset requires aggregating requirements from multiple sources (foundations, layouts, schemas sections).

Evidence

• [other] Does the Rust implementation of mzPeak produce files that conform to all mandatory fields and structural elements defined in the published mzPeak specification?: "Does the Rust implementation of mzPeak produce files that conform to all mandatory fields and structural elements defined in the published mzPeak specification?"
• [other] A draft mzPeak specification document is publicly available at https://hupo-psi.github.io/mzPeak-specification/ against which Rust implementation output can be validated for compliance.: "A draft mzPeak specification document is publicly available at https://hupo-psi.github.io/mzPeak-specification/ against which Rust implementation output can be validated for compliance."
• [readme] mzPeak is a archive of multiple Parquet files, stored directly in an uncompressed ZIP archive. Each Parquet file describes a different facet of the stored mass spectrometry run.: "mzPeak is a archive of multiple Parquet files, stored directly in an uncompressed ZIP archive. Each Parquet file describes a different facet of the stored mass spectrometry run."
• [readme] Components of an mzPeak archive: mzpeak_index.json, spectra_metadata.parquet, spectra_data.parquet, spectra_peaks.parquet (optional), chromatograms_metadata.parquet, chromatograms_data.parquet: "mzpeak_index.json: Definition of the files present in the archive, encoded as JSON... spectra_metadata.parquet: Spectrum level metadata... spectra_data.parquet: Spectrum signal data..."
• [readme] mzPeak file-level metadata, including descriptions of the file's contents, the instrumentation, software, and data transformation pipeline are stored in the Parquet metadata segment as JSON documents.: "mzPeak file-level metadata, including descriptions of the file's contents, the instrumentation, software, and data transformation pipeline are stored in the Parquet metadata segment as JSON documents."
• [readme] This is a work in progress, no stability is guaranteed at this point.: "This is a work in progress, no stability is guaranteed at this point."