metadata-structure-transformation
Summary
Transform tabular experimental metadata (Excel/CSV) into clean, standardized JSONized intermediate format using a tagging system, then convert to repository-compliant output formats (e.g., mwTab for Metabolomics Workbench). This skill bridges raw data entry and repository submission by enforcing schema compliance and enabling format-agnostic validation.
When to use
You have raw tabular experimental metadata (mass spectrometry or NMR sample descriptions, sample-to-treatment mappings, instrument parameters, etc.) that needs to be deposited into a structured online repository like Metabolomics Workbench, but the raw format does not conform to the repository's expected schema. Use this skill when you need to move from spreadsheet-based data entry to a standardized, validatable, and format-convertible representation.
When NOT to use
- Input is already in the target repository format (e.g., already a valid mwTab file) — use direct validation instead.
- Data is already properly structured JSON conforming to Experiment Description Specification and your target format schema — skip directly to conversion.
- Tabular data is raw instrument output (e.g., raw mass spectrometry data files, unprocessed spectral data) rather than experimental metadata or sample annotations — use instrument-specific parsing tools instead.
Inputs
- Tabular data file (Excel .xlsx or CSV format) with raw experimental metadata
- Tagging specification defining semantic markup for tabular columns
- Protocol Dependent Schema (JSON Schema) describing project-specific constraints
- Format-specific schema (e.g., mwTab schema) for target repository
- Conversion directives (JSON) specifying output transformations (e.g., collate, filter, rename operations)
Outputs
- Intermediate JSON file conforming to Experiment Description Specification
- Validated intermediate JSON with error/warning report
- Repository-compliant output file (e.g., mwTab format for Metabolomics Workbench)
How to apply
The workflow follows three steps: (1) Add a tagging layer to your tabular data (manual or automated) that marks columns with semantic meaning (e.g., sample ID, metabolite name, intensity value). (2) Run the extract command to parse the tagged tabular data into an intermediate JSON representation conforming to the Experiment Description Specification schema. (3) Validate the extracted JSON using the validate command against the Protocol Dependent Schema, custom project schemas, and format-specific schemas (e.g., mwTab schema) to catch errors. (4) Apply conversion directives (such as collate='assignment' for grouping records by metabolite assignment) using the convert command to transform the JSON into the final repository format. The rationale is that JSON serves as a format-agnostic interchange layer: once data is correctly extracted and validated in JSON form, it can be reliably converted to any supported output format without re-entering or re-validating the raw data.
Related tools
- MESSES (Metadata from Experimental SpreadSheets Extraction System) (Core command-line tool for executing extract, validate, and convert commands on tabular and JSON data) — https://github.com/MoseleyBioinformaticsLab/messes
- jsonschema (Validates intermediate and output JSON against Experiment Description Specification, Protocol Dependent Schema, and format-specific schemas) — https://pypi.org/project/jsonschema/
- Python (Runtime language for MESSES package and programmatic extension of conversion logic)
Examples
messes extract your_data.csv --output your_data.json && messes validate json your_data.json --pds your_schema.json --format mwTab && messes convert mwTab your_data.json output_data
Evaluation signals
- Extracted JSON file is valid against the Experiment Description Specification schema (confirmed by
validate command with zero errors).
- When conversion directives are applied (e.g.,
collate='assignment'), measurement records are correctly grouped: verify that all records sharing the same metabolite assignment are merged into a single dictionary containing the metabolite name and a merged sample intensity dictionary, with no duplicates or lost records.
- Validation report from
validate command shows zero errors against Protocol Dependent Schema and format-specific schema (e.g., mwTab schema).
- Final output file can be parsed by the target repository's ingestion tools without schema violations or format errors.
- Spot-check that sample intensity values, metabolite names, and assignment IDs are preserved (not truncated, corrupted, or reordered) during extraction and conversion steps.
Limitations
- Requires manual or automatable tagging of raw tabular data before extraction; untagged or inconsistently tagged columns will be missed or cause extraction errors.
- The tagging system and conversion directives have a steep initial learning curve; proper understanding of the Experiment Description Specification and Protocol Dependent Schema is essential before use.
- Conversion to non-mwTab formats is not yet supported by the library (only mwTab is currently documented as a supported output format for Metabolomics Workbench).
- Complex data transformations beyond simple collation, filtering, and renaming may require custom JSON-to-JSON directives or programmatic extension of the MESSES library.
- File format support is limited to tabular input (Excel, CSV); other raw data formats (e.g., binary instrument files, HDF5) must be converted to tabular form before extraction.
Evidence
- [readme] MESSES (Metadata from Experimental SpreadSheets Extraction System) is a Python package that facilitates the conversion of tabular data into other formats.: "MESSES (Metadata from Experimental SpreadSheets Extraction System) is a Python package that facilitates the conversion of tabular data into other formats."
- [readme] Simply add a layer of tags to any tabular data and MESSES can transform it into an intermediate JSON representation and then convert it to any of the supported formats.: "Simply add a layer of tags to any tabular data and MESSES can transform it into an intermediate JSON representation and then convert it to any of the supported formats."
- [readme] MESSES breaks up the process into 3 steps: extract, validate, and convert.: "MESSES breaks up the process into 3 steps: extract, validate, and convert."
- [readme] The extraction step adds a layer of tags to your raw tabular data, which may be automatable, and then extracts it into a JSONized form that it is more interoperable and more standardized.: "The extraction step adds a layer of tags to your raw tabular data, which may be automatable, and then extracts it into a JSONized form that it is more interoperable and more standardized."
- [readme] The validation step ensures the data that was extracted is valid against the Experiment Description Specification, the Protocol Dependent Schema, any additional JSON schema you wish to provide, and a built in schema specific for the format you wish to convert to.: "The validation step ensures the data that was extracted is valid against the Experiment Description Specification, the Protocol Dependent Schema, any additional JSON schema you wish to provide, and a"
- [other] The collate directive groups four measurement records into two dictionaries by assignment, with each dictionary containing the metabolite name and intensity values from all samples sharing that assignment.: "The collate directive groups four measurement records into two dictionaries by assignment, with each dictionary containing the metabolite name and intensity values from all samples sharing that"
- [intro] To support the JSON-to-JSON conversion a relatively simple set of directives were developed: "To support the JSON-to-JSON conversion a relatively simple set of directives were developed"
1---2name: metadata-structure-transformation-33description: Use when you have raw tabular experimental metadata (mass spectrometry4 or NMR sample descriptions, sample-to-treatment mappings, instrument parameters,5 etc.) that needs to be deposited into a structured online repository like Metabolomics6 Workbench, but the raw format does not conform to the.7license: CC-BY-4.08metadata:9 source: https://skillmd.com/skills/holobiomicslab/metadata-structure-transformation-310 edam_operation: http://edamontology.org/operation_375011 edam_topics:12 - http://edamontology.org/topic_339113 - http://edamontology.org/topic_317214 tools:15 - Python16 - jsonschema17 - MESSES (Metadata from Experimental SpreadSheets Extraction System)18 techniques:19 - NMR20 license_tier: open21 provenance_tier: literature22derived_from:23- doi: 10.3390/metabo1307084224 title: messes25- doi: 10.3390/metabo1103016326 title: ''27evidence_spans:28- MESSES (Metadata from Experimental SpreadSheets Extraction System) is a Python package29- utilizing `JSON Schema <https://json-schema.org/understanding-json-schema/>`_ (`jsonschema30 <https://pypi.org/project/jsonschema/>`_)31claims: []32provenance:33 collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v234 assembled_by: scripts/collect_metabolomics_collection.py35 sources:36 - build: coll_messes37 doi: 10.3390/metabo1307084238 title: messes39 dedup_kept_from: coll_messes40schema_version: 0.2.041attribution:42 generator: AgenticScienceBuilder43 original_doi: 10.3390/metabo1307084244 all_source_dois:45 - 10.3390/metabo1307084246 - 10.3390/metabo1103016347 zenodo_doi: 10.5281/zenodo.2079402748 curators: []49 promoter: Louis-Félix Nothias50 sponsor: CNRS & Université Côte d'Azur51---5253# metadata-structure-transformation5455## Summary5657Transform tabular experimental metadata (Excel/CSV) into clean, standardized JSONized intermediate format using a tagging system, then convert to repository-compliant output formats (e.g., mwTab for Metabolomics Workbench). This skill bridges raw data entry and repository submission by enforcing schema compliance and enabling format-agnostic validation.5859## When to use6061You have raw tabular experimental metadata (mass spectrometry or NMR sample descriptions, sample-to-treatment mappings, instrument parameters, etc.) that needs to be deposited into a structured online repository like Metabolomics Workbench, but the raw format does not conform to the repository's expected schema. Use this skill when you need to move from spreadsheet-based data entry to a standardized, validatable, and format-convertible representation.6263## When NOT to use6465- Input is already in the target repository format (e.g., already a valid mwTab file) — use direct validation instead.66- Data is already properly structured JSON conforming to Experiment Description Specification and your target format schema — skip directly to conversion.67- Tabular data is raw instrument output (e.g., raw mass spectrometry data files, unprocessed spectral data) rather than experimental metadata or sample annotations — use instrument-specific parsing tools instead.6869## Inputs7071- Tabular data file (Excel .xlsx or CSV format) with raw experimental metadata72- Tagging specification defining semantic markup for tabular columns73- Protocol Dependent Schema (JSON Schema) describing project-specific constraints74- Format-specific schema (e.g., mwTab schema) for target repository75- Conversion directives (JSON) specifying output transformations (e.g., collate, filter, rename operations)7677## Outputs7879- Intermediate JSON file conforming to Experiment Description Specification80- Validated intermediate JSON with error/warning report81- Repository-compliant output file (e.g., mwTab format for Metabolomics Workbench)8283## How to apply8485The workflow follows three steps: (1) Add a tagging layer to your tabular data (manual or automated) that marks columns with semantic meaning (e.g., sample ID, metabolite name, intensity value). (2) Run the `extract` command to parse the tagged tabular data into an intermediate JSON representation conforming to the Experiment Description Specification schema. (3) Validate the extracted JSON using the `validate` command against the Protocol Dependent Schema, custom project schemas, and format-specific schemas (e.g., mwTab schema) to catch errors. (4) Apply conversion directives (such as `collate='assignment'` for grouping records by metabolite assignment) using the `convert` command to transform the JSON into the final repository format. The rationale is that JSON serves as a format-agnostic interchange layer: once data is correctly extracted and validated in JSON form, it can be reliably converted to any supported output format without re-entering or re-validating the raw data.8687## Related tools8889- **MESSES (Metadata from Experimental SpreadSheets Extraction System)** (Core command-line tool for executing extract, validate, and convert commands on tabular and JSON data) — https://github.com/MoseleyBioinformaticsLab/messes90- **jsonschema** (Validates intermediate and output JSON against Experiment Description Specification, Protocol Dependent Schema, and format-specific schemas) — https://pypi.org/project/jsonschema/91- **Python** (Runtime language for MESSES package and programmatic extension of conversion logic)9293## Examples9495```96messes extract your_data.csv --output your_data.json && messes validate json your_data.json --pds your_schema.json --format mwTab && messes convert mwTab your_data.json output_data97```9899## Evaluation signals100101- Extracted JSON file is valid against the Experiment Description Specification schema (confirmed by `validate` command with zero errors).102- When conversion directives are applied (e.g., `collate='assignment'`), measurement records are correctly grouped: verify that all records sharing the same metabolite assignment are merged into a single dictionary containing the metabolite name and a merged sample intensity dictionary, with no duplicates or lost records.103- Validation report from `validate` command shows zero errors against Protocol Dependent Schema and format-specific schema (e.g., mwTab schema).104- Final output file can be parsed by the target repository's ingestion tools without schema violations or format errors.105- Spot-check that sample intensity values, metabolite names, and assignment IDs are preserved (not truncated, corrupted, or reordered) during extraction and conversion steps.106107## Limitations108109- Requires manual or automatable tagging of raw tabular data before extraction; untagged or inconsistently tagged columns will be missed or cause extraction errors.110- The tagging system and conversion directives have a steep initial learning curve; proper understanding of the Experiment Description Specification and Protocol Dependent Schema is essential before use.111- Conversion to non-mwTab formats is not yet supported by the library (only mwTab is currently documented as a supported output format for Metabolomics Workbench).112- Complex data transformations beyond simple collation, filtering, and renaming may require custom JSON-to-JSON directives or programmatic extension of the MESSES library.113- File format support is limited to tabular input (Excel, CSV); other raw data formats (e.g., binary instrument files, HDF5) must be converted to tabular form before extraction.114115## Evidence116117- [readme] MESSES (Metadata from Experimental SpreadSheets Extraction System) is a Python package that facilitates the conversion of tabular data into other formats.: "MESSES (Metadata from Experimental SpreadSheets Extraction System) is a Python package that facilitates the conversion of tabular data into other formats."118- [readme] Simply add a layer of tags to any tabular data and MESSES can transform it into an intermediate JSON representation and then convert it to any of the supported formats.: "Simply add a layer of tags to any tabular data and MESSES can transform it into an intermediate JSON representation and then convert it to any of the supported formats."119- [readme] MESSES breaks up the process into 3 steps: extract, validate, and convert.: "MESSES breaks up the process into 3 steps: extract, validate, and convert."120- [readme] The extraction step adds a layer of tags to your raw tabular data, which may be automatable, and then extracts it into a JSONized form that it is more interoperable and more standardized.: "The extraction step adds a layer of tags to your raw tabular data, which may be automatable, and then extracts it into a JSONized form that it is more interoperable and more standardized."121- [readme] The validation step ensures the data that was extracted is valid against the Experiment Description Specification, the Protocol Dependent Schema, any additional JSON schema you wish to provide, and a built in schema specific for the format you wish to convert to.: "The validation step ensures the data that was extracted is valid against the Experiment Description Specification, the Protocol Dependent Schema, any additional JSON schema you wish to provide, and a"122- [other] The collate directive groups four measurement records into two dictionaries by assignment, with each dictionary containing the metabolite name and intensity values from all samples sharing that assignment.: "The collate directive groups four measurement records into two dictionaries by assignment, with each dictionary containing the metabolite name and intensity values from all samples sharing that"123- [intro] To support the JSON-to-JSON conversion a relatively simple set of directives were developed: "To support the JSON-to-JSON conversion a relatively simple set of directives were developed"