microbiome-metabolome-data-preprocessing
Summary
Preparation and normalization of paired microbiome and metabolome feature tables for downstream prediction modeling, including low-abundance filtering, compositional transformations (CLR or relative abundance), and feature quality control. This skill ensures that heterogeneous omics measurements are made commensurable and that rare features do not dominate the learning signal.
When to use
When starting with raw paired microbiome (16S rRNA, metagenomic taxonomic or functional features) and metabolome (LC-MS/MS, NMR) count tables from the same biospecimens, and planning to train prediction models or co-abundance networks. Apply this skill before neural network training, module clustering, or any cross-validated evaluation on microbiome-metabolome datasets.
When NOT to use
- Input feature tables are already normalized and filtered (e.g., already CLR-transformed or rarefied)
- Microbiome and metabolome measurements come from different or unpaired biospecimens
- Analysis goal does not require training a predictive model; descriptive alpha/beta diversity analysis may not need CLR transformation
Inputs
- raw microbiome count table (samples × microbes/features, integer or relative abundance format)
- raw metabolome count table (samples × metabolites/features, LC-MS/MS or NMR intensity format)
- optional: sample metadata (e.g., diagnosis labels, cohort identifiers)
Outputs
- filtered microbiome feature table (samples × microbes, after <10% prevalence removal)
- filtered metabolome feature table (samples × metabolites, after <10% prevalence removal)
- transformation metadata (normalization type: CLR or RA, pseudocount value if CLR, any filtering thresholds applied)
How to apply
Load raw paired microbiome and metabolome count tables (samples × features format). First, remove any feature (microbial or metabolite) present in fewer than 10% of samples to eliminate sparse, likely artifactual features. Second, apply compositional transformation: either centered log-ratio (CLR) with pseudocount +1, or relative abundance (RA) normalization. The choice between CLR and RA affects downstream correlation estimates; CLR is recommended for datasets with compositional structure (cystic fibrosis, soil), while RA may suffice for well-characterized cohorts like IBD PRISM. Document the transformation applied, as it influences background distribution generation and well-predicted metabolite thresholds. Return filtered and normalized feature tables (same row/column structure) and record preprocessing parameters for reproducibility in external validation.
Related tools
- scikit-learn (Preprocessing and normalization of feature tables; feature selection and filtering utilities)
- SciPy (Centered log-ratio transformation via logarithmic and geometric mean operations)
- Pandas (Loading, filtering, and reshaping CSV/TSV count tables and metadata)
- NumPy (Array operations for prevalence filtering and composition-altering transformations)
Examples
python MiMeNet_train.py -micro data/IBD/microbiome_PRISM.csv -metab data/IBD/metabolome_PRISM.csv -micro_norm None -metab_norm CLR
Evaluation signals
- Verify that no feature in output tables has <10% prevalence (sample coverage check)
- Confirm that CLR-transformed features have mean ≈ 0 across samples; RA features sum to 1.0 per sample
- Check that row and column counts match between input and output (only features below threshold removed)
- If CLR applied, verify that log-transformed denominators (geometric mean) do not yield infinite or NaN values
- Reproduce output on held-out samples using the same transformation parameters; identical results confirm deterministic preprocessing
Limitations
- The 10% prevalence threshold is heuristic and may remove true rare features; datasets with sparse, true low-abundance species may require lower cutoffs
- CLR transformation with pseudocount +1 can bias small counts; alternative pseudocounts (e.g., half-minimum nonzero, Laplace) may be preferable for sparse data
- Choice of CLR vs. RA normalization affects subsequent model performance and threshold cutoffs; no universal guidance provided; empirical comparison (task_005) is recommended
- Paired sample assumption is critical; if microbiome and metabolome are from different biospecimens or time-points, compositional methods may mislead
- For longitudinal or hierarchical study designs (e.g., repeated measures, biocrust time-series), batch effects and temporal autocorrelation are not addressed in this preprocessing step
Evidence
- [methods] Remove features present in <10% of samples: "Any input or output feature that is present in less than 10% of samples was removed"
- [methods] CLR transformation with pseudocount +1: "centered log-ratio (CLR) transformation with pseudocount +1"
- [methods] Relative abundance normalization alternative: "relative abundance (RA) normalization"
- [abstract] CLR vs RA effects on downstream prediction: "prediction correlations were comparable between CLR and relative abundance transformations, while CLR showed increased correlations in cystic fibrosis and soil datasets"
- [methods] Dataset-specific preprocessing parameters: "centered log-ratio transformation (with pseudocount of 1; exception: IBD PRISM microbes in relative abundance)"
- [methods] Preprocessing prior to neural network training: "Load and preprocess the three paired microbiome-metabolome datasets"
1---2name: microbiome-metabolome-data-preprocessing-33description: Use when when starting with raw paired microbiome (16S rRNA, metagenomic4 taxonomic or functional features) and metabolome (LC-MS/MS, NMR) count tables from5 the same biospecimens, and planning to train prediction models or co-abundance networks.6license: CC-BY-4.07metadata:8 source: https://skillmd.com/skills/holobiomicslab/microbiome-metabolome-data-preprocessing-39 edam_operation: http://edamontology.org/operation_343410 edam_topics:11 - http://edamontology.org/topic_317412 - http://edamontology.org/topic_063713 - http://edamontology.org/topic_340714 tools:15 - MiMeNet16 - scikit-learn (MLPRegressor)17 - ADAM optimizer18 - ReLU activation19 - NumPy20 - Elastic Net21 - TensorFlow or PyTorch22 - scikit-learn23 - Seaborn24 - Python25 - SciPy26 - Pandas27 techniques:28 - LC-MS29 - NMR30 license_tier: open31 provenance_tier: literature32derived_from:33- doi: 10.1371/journal.pcbi.100902134 title: MiMeNet35evidence_spans:36- MiMeNet (Microbiome-Metabolome Network), a multi-layer perceptron (MLPNN)37- MiMeNet uses paired microbiome and metabolome data for model training. Microbiome38 abundance features (green) are used to train a neural network to predict metabolite39 abundance features (blue).40- An MLPNN model is composed of multiple fully connected hidden layers composed of41 perceptrons42- Canonical correlation analysis models were implemented using Python's scikit-learn43 package.44- MiMeNet was trained using the ADAM optimizer and the mean squared error (MSE) loss45 function.46- In MiMeNet, φ is set as the rectified linear unit (ReLU). We selected this activation47 function since previous studies have shown that it is resilient to the problems48 of exploding and vanishing49claims: []50provenance:51 collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v252 assembled_by: scripts/collect_metabolomics_collection.py53 sources:54 - build: coll_mimenet_cq55 doi: 10.1371/journal.pcbi.100902156 title: MiMeNet57 dedup_kept_from: coll_mimenet_cq58schema_version: 0.2.059attribution:60 generator: AgenticScienceBuilder61 original_doi: 10.1371/journal.pcbi.100902162 all_source_dois:63 - 10.1371/journal.pcbi.100902164 zenodo_doi: 10.5281/zenodo.2079402765 curators: []66 promoter: Louis-Félix Nothias67 sponsor: CNRS & Université Côte d'Azur68---6970# microbiome-metabolome-data-preprocessing7172## Summary7374Preparation and normalization of paired microbiome and metabolome feature tables for downstream prediction modeling, including low-abundance filtering, compositional transformations (CLR or relative abundance), and feature quality control. This skill ensures that heterogeneous omics measurements are made commensurable and that rare features do not dominate the learning signal.7576## When to use7778When starting with raw paired microbiome (16S rRNA, metagenomic taxonomic or functional features) and metabolome (LC-MS/MS, NMR) count tables from the same biospecimens, and planning to train prediction models or co-abundance networks. Apply this skill before neural network training, module clustering, or any cross-validated evaluation on microbiome-metabolome datasets.7980## When NOT to use8182- Input feature tables are already normalized and filtered (e.g., already CLR-transformed or rarefied)83- Microbiome and metabolome measurements come from different or unpaired biospecimens84- Analysis goal does not require training a predictive model; descriptive alpha/beta diversity analysis may not need CLR transformation8586## Inputs8788- raw microbiome count table (samples × microbes/features, integer or relative abundance format)89- raw metabolome count table (samples × metabolites/features, LC-MS/MS or NMR intensity format)90- optional: sample metadata (e.g., diagnosis labels, cohort identifiers)9192## Outputs9394- filtered microbiome feature table (samples × microbes, after <10% prevalence removal)95- filtered metabolome feature table (samples × metabolites, after <10% prevalence removal)96- transformation metadata (normalization type: CLR or RA, pseudocount value if CLR, any filtering thresholds applied)9798## How to apply99100Load raw paired microbiome and metabolome count tables (samples × features format). First, remove any feature (microbial or metabolite) present in fewer than 10% of samples to eliminate sparse, likely artifactual features. Second, apply compositional transformation: either centered log-ratio (CLR) with pseudocount +1, or relative abundance (RA) normalization. The choice between CLR and RA affects downstream correlation estimates; CLR is recommended for datasets with compositional structure (cystic fibrosis, soil), while RA may suffice for well-characterized cohorts like IBD PRISM. Document the transformation applied, as it influences background distribution generation and well-predicted metabolite thresholds. Return filtered and normalized feature tables (same row/column structure) and record preprocessing parameters for reproducibility in external validation.101102## Related tools103104- **scikit-learn** (Preprocessing and normalization of feature tables; feature selection and filtering utilities)105- **SciPy** (Centered log-ratio transformation via logarithmic and geometric mean operations)106- **Pandas** (Loading, filtering, and reshaping CSV/TSV count tables and metadata)107- **NumPy** (Array operations for prevalence filtering and composition-altering transformations)108109## Examples110111```112python MiMeNet_train.py -micro data/IBD/microbiome_PRISM.csv -metab data/IBD/metabolome_PRISM.csv -micro_norm None -metab_norm CLR113```114115## Evaluation signals116117- Verify that no feature in output tables has <10% prevalence (sample coverage check)118- Confirm that CLR-transformed features have mean ≈ 0 across samples; RA features sum to 1.0 per sample119- Check that row and column counts match between input and output (only features below threshold removed)120- If CLR applied, verify that log-transformed denominators (geometric mean) do not yield infinite or NaN values121- Reproduce output on held-out samples using the same transformation parameters; identical results confirm deterministic preprocessing122123## Limitations124125- The 10% prevalence threshold is heuristic and may remove true rare features; datasets with sparse, true low-abundance species may require lower cutoffs126- CLR transformation with pseudocount +1 can bias small counts; alternative pseudocounts (e.g., half-minimum nonzero, Laplace) may be preferable for sparse data127- Choice of CLR vs. RA normalization affects subsequent model performance and threshold cutoffs; no universal guidance provided; empirical comparison (task_005) is recommended128- Paired sample assumption is critical; if microbiome and metabolome are from different biospecimens or time-points, compositional methods may mislead129- For longitudinal or hierarchical study designs (e.g., repeated measures, biocrust time-series), batch effects and temporal autocorrelation are not addressed in this preprocessing step130131## Evidence132133- [methods] Remove features present in <10% of samples: "Any input or output feature that is present in less than 10% of samples was removed"134- [methods] CLR transformation with pseudocount +1: "centered log-ratio (CLR) transformation with pseudocount +1"135- [methods] Relative abundance normalization alternative: "relative abundance (RA) normalization"136- [abstract] CLR vs RA effects on downstream prediction: "prediction correlations were comparable between CLR and relative abundance transformations, while CLR showed increased correlations in cystic fibrosis and soil datasets"137- [methods] Dataset-specific preprocessing parameters: "centered log-ratio transformation (with pseudocount of 1; exception: IBD PRISM microbes in relative abundance)"138- [methods] Preprocessing prior to neural network training: "Load and preprocess the three paired microbiome-metabolome datasets"