Metabolomics Data Formatting

metabolomics-data-formatting

Summary

Format and quality-control processed LCMS metabolomics feature tables to conform to a shared schema with standardized column headers, metadata structure, and integrity checks. Formation is a standalone BMXP module that executes this final pipeline step before data release or downstream analysis.

When to use

After running feature clustering (Gravity) or drift correction (Blueshift) on LCMS data, when you have a processed feature table and need to standardize its structure, validate metadata completeness, enforce missing-value thresholds, and generate a QC report documenting pass/fail status before sharing or analyzing the data.

When NOT to use

• Input is raw .raw or .mzML files — use Chroma to read instrument files first
• Data has not yet been aligned across datasets — run Eclipse before Formation
• Feature redundancy has not been resolved — run Gravity to cluster correlated features before formatting

Inputs

• Processed feature table from Gravity or Blueshift (feature metadata + feature abundances)
• Injection metadata table with injection_id index
• Sample metadata table with broad_id index

Outputs

• Formatted feature metadata table (bmxp.FMDATA schema)
• Formatted feature abundances pivot table (Compound_ID × injection_id)
• Formatted injection metadata table (bmxp.IMDATA schema)
• Formatted sample metadata table (bmxp.SMDATA schema)
• Formation QC report with pass/fail status for each check

How to apply

Load the processed feature table (output from Blueshift or Gravity) in the appropriate metabolomics data format. Apply Formation-specific formatting rules to standardize feature annotations, metadata structure, and column naming conventions to match the shared BMXP schema (Compound_ID, RT, MZ, Intensity, Method, and annotation fields for features; injection_id, broad_id, injection_type, batches for injections; broad_id and arbitrary metadata columns for samples). Execute Formation's quality control checks on feature integrity (e.g., RT and MZ validity), missing-value thresholds across the feature abundance pivot table, and metadata completeness (all required columns present and non-null). Generate both a formatted output file and a detailed QC report documenting pass/fail status for each check. Validate that the output meets Formation acceptance criteria (schema compliance, QC pass flags) before finalizing for downstream use.

Related tools

• bmxp (Python package containing Formation module and shared schema definitions) — https://github.com/broadinstitute/bmxp
• Gravity (Upstream module that clusters redundant LCMS features; output is input to Formation) — https://github.com/broadinstitute/bmxp/blob/main/bmxp/gravity/readme.md
• Blueshift (Upstream module that performs drift correction; output is input to Formation) — https://github.com/broadinstitute/bmxp/blob/main/bmxp/blueshift/readme.md
• Formation (Standalone BMXP module that applies formatting rules and QC checks) — https://github.com/broadinstitute/bmxp/blob/main/bmxp/formation/readme.md

Evaluation signals

• All feature metadata columns match bmxp.FMDATA schema (Compound_ID, RT, MZ, Intensity, Method, Annotation_ID, Adduct, Metabolite, etc.)
• All injection metadata columns match bmxp.IMDATA schema (injection_id, broad_id, injection_type, batches, etc.)
• All sample metadata columns match bmxp.SMDATA schema (broad_id as index, with arbitrary biospecimen metadata)
• Feature abundance table is a valid pivot table with shape (n_features, n_injections) indexed by Compound_ID × injection_id with no unexpected nulls
• QC report shows pass status for feature integrity, missing-value thresholds, and metadata completeness checks

Limitations

• Formation requires input data to already be processed through upstream modules (Chroma, Eclipse, Gravity, Blueshift); it does not handle raw instrument files or unaligned datasets
• The shared schema column headers can be customized globally before runtime but must be consistent across all modules used in the pipeline
• Formation does not perform feature annotation or metabolite identification; annotation fields (Annotation_ID, Metabolite) are expected to be populated upstream or externally
• QC checks are based on predefined thresholds for missing values and metadata completeness; highly non-standard datasets may fail checks without custom configuration

Evidence

• [other] Formation is a standalone module in the BMXP pipeline that performs formatting and final quality control on processed LCMS feature data.: "Formation is a standalone module in the BMXP pipeline that performs formatting and final quality control on processed LCMS feature data."
• [other] Load the processed feature table (output from Blueshift or Gravity) in the appropriate metabolomics data format. Apply Formation-specific formatting rules to standardize feature annotations, metadata structure, and column naming conventions.: "Load the processed feature table (output from Blueshift or Gravity) in the appropriate metabolomics data format. Apply Formation-specific formatting rules to standardize feature annotations, metadata"
• [other] Execute quality control checks on feature integrity, missing-value thresholds, and metadata completeness. Generate a formatted output file and a QC report documenting pass/fail status for each check.: "Execute quality control checks on feature integrity, missing-value thresholds, and metadata completeness. Generate a formatted output file and a QC report documenting pass/fail status for each check."
• [readme] All BMXP modules use a shared schema and file formats with our prefered columns headers. These files are (along with their labels): Feature Metadata bmxp.FMDATA - Describes the feature. Index default is Compound_ID: "All BMXP modules use a shared schema and file formats with our prefered columns headers. These files are (along with their labels): Feature Metadata bmxp.FMDATA - Describes the feature. Index"
• [readme] Feature Abundances - Pivot table of Feature x Injection (Compound_ID x injection_id) containing the abundances.: "Feature Abundances - Pivot table of Feature x Injection (Compound_ID x injection_id) containing the abundances."
• [readme] Each tool is meant to be a standalone module that performs a step in our processing pipeline. They are written in Python and C, and designed to be perfomant and cloud-compatible.: "Each tool is meant to be a standalone module that performs a step in our processing pipeline. They are written in Python and C, and designed to be perfomant and cloud-compatible."