Metadata Harmonization Standardization

metadata-harmonization-standardization

Summary

Harmonize and standardize MS/MS spectral metadata fields (compound identifiers, adduct annotations, collision energies, instrument types) to a common schema after preprocessing. This ensures consistency across heterogeneous spectral library sources and enables reliable downstream analysis and cross-library comparisons.

When to use

When you have preprocessed MS/MS spectra from multiple source repositories or instruments with inconsistent metadata field naming, formats, or values (e.g., mixed adduct notations like '[M+H]+' vs '[M+H]⁺', variable instrument type strings, or non-standard collision energy units). Apply this skill after spectral format standardization and quality filtering but before duplicate detection and final library export.

When NOT to use

• Input spectra are already part of a single, internally consistent spectral library (e.g., a single instrument's native format with uniform metadata conventions); harmonization will add unnecessary processing overhead.
• Metadata fields are already in a validated, standardized schema and preprocessing/filtering has not introduced new inconsistencies or duplicates.
• The analysis goal requires preservation of source-specific metadata variations (e.g., historical instrument-specific collision energy notation) rather than normalization to a common standard.

Inputs

• Preprocessed MS/MS spectra in MGF format with source metadata fields
• Metadata CSV files extracted from MGF headers
• Repository mapping configuration files defining standard schema (adduct list, instrument name mappings, collision energy unit conversions)
• matchms filter YAML configuration files (library_cleaning_1.yaml, library_cleaning_2.yaml)

Outputs

• Harmonized metadata CSV files with standardized field values
• Harmonized MGF files with corrected and unified metadata annotations
• Duplicate candidate list with pairwise cosine similarity scores (numpy format)
• Final curated spectral library in standardized output format (mzML, mzTab, or MGF)

How to apply

Execute the Spectraverse harmonization pipeline in stages: (1) Load preprocessed MGF files and associated metadata via mgf2csv conversion; (2) Apply standardization scripts (step2-3_standardization.py for SMILES, step3-2_adduct-rem.py for adduct correction, step3-7_metadata.py for instrument and collision energy fields) that map non-standard values to a repository-defined common schema; (3) Run matchms library cleaning filters (library_cleaning_1.yaml and library_cleaning_2.yaml) to repair metadata coherence after each stage; (4) Validate harmonized metadata against schema requirements (e.g., adduct field contains only approved adduct strings, collision energy values are numeric and within expected ranges); (5) Identify and flag or remove duplicate spectra based on harmonized metadata combinations using cosine similarity thresholds (pairwise cosine scores calculated in step3-4); (6) Export final harmonized library in standardized format (mzML, mzTab, or MGF with corrected metadata).

Related tools

• matchms (Repairs and validates metadata consistency; applies standardized cleaning filters after each harmonization stage to correct incoherent annotations and remove metadata errors) — https://github.com/matchms/matchms
• SpectralEntropy (Calculates cosine similarity scores for duplicate spectrum detection based on harmonized spectra) — https://github.com/YuanyueLi/SpectralEntropy.git
• spectraverse-analysis pipeline scripts (Orchestrates multi-stage harmonization: mgf2csv, step2-3_standardization.py (SMILES), step3-2_adduct-rem.py (adduct correction), step3-7_metadata.py (instrument/collision energy standardization), step3-5_uniq-select.py (duplicate removal)) — https://github.com/skinniderlab/spectraverse-analysis

Examples

python run_steps.py --config config/config_step3.json

Evaluation signals

• All adduct annotations conform to a predefined controlled vocabulary (e.g., no malformed or undeclared adduct strings; step3-2 reports count of corrections applied).
• Instrument type field contains only standardized instrument identifiers; no remaining free-text variants or abbreviations outside the approved schema.
• Collision energy values are numeric, fall within expected ranges for the instrument type, and use consistent units (e.g., all eV, all normalized 0–100 scale).
• Pairwise cosine similarity scores are computed for all candidate duplicate spectra pairs; duplicates identified by score threshold (implicit in step3-5_uniq-select.py) are removed, reducing redundancy.
• Final library export validates against schema: no null/missing required metadata fields, no spectra with identical fragment m/z intensity patterns (removed in step2-2), no spectra with all fragment m/z > precursor m/z (removed in step2-7).

Limitations

• Harmonization depends on the completeness and accuracy of the predefined schema mappings; metadata fields not covered by the mapping rules may remain non-standard or be lost.
• matchms repair functions may fail or produce incorrect mappings for highly corrupted or non-standard metadata entries; manual curation may be required for outliers.
• Duplicate detection via cosine similarity is heuristic-dependent; setting the cosine threshold too high may retain duplicates, while too low may remove genuine spectral variants.
• No changelog is available in the repository to track schema evolution or previous harmonization decisions, limiting reproducibility and version control of metadata changes.

Evidence

• [other] Harmonize metadata fields (compound identifiers, adduct annotations, collision energies, instrument types) to common schema using repository mapping functions.: "Harmonize metadata fields (compound identifiers, adduct annotations, collision energies, instrument types) to common schema using repository mapping functions."
• [intro] code used to preprocess and harmonize MS/MS spectra and their associated metadata in the compilation of Spectraverse.: "code used to preprocess and harmonize MS/MS spectra and their associated metadata in the compilation of Spectraverse."
• [readme] step3-7_metadata.py: Standardization of instrument and collision energy fields, and finalizing the metadata: "step3-7_metadata.py: Standardization of instrument and collision energy fields, and finalizing the metadata"
• [readme] step3-2_adduct-rem.py: Removal and correction of unwanted adducts: "step3-2_adduct-rem.py: Removal and correction of unwanted adducts"
• [readme] At this point, matchms is run to repair the metadata associated with each spectrum.: "At this point, matchms is run to repair the metadata associated with each spectrum."
• [readme] step3-5_uniq-select.py: Removal of duplicate spectra based on pairwise cosine scores: "step3-5_uniq-select.py: Removal of duplicate spectra based on pairwise cosine scores"