Consensus-clustering module construction
Summary
Group microbes and metabolites with similar interaction patterns into functional modules by biclustering normalized feature attribution score matrices, using consensus clustering with data-driven k* selection. This recovers the latent structure of microbe-metabolite interaction networks and enables module-level biomarker discovery.
When to use
After training a neural network model on paired microbiome-metabolome data and extracting microbe-metabolite feature attribution scores for significant interactions (e.g., at the 97.5th percentile threshold). Use this skill when you want to group co-interacting microbes and metabolites to illuminate functional relationships, especially when individual microbial or metabolite annotations are sparse or when you seek to validate predicted modules against known disease states.
When NOT to use
- Attribution score matrix contains <10 significant microbe-metabolite interactions; consensus clustering requires sufficient data density to recover meaningful structure.
- Background distribution was not generated (i.e., no shuffled cross-validation performed); modules would lack statistical grounding and risk spurious clustering.
- Microbes or metabolites have already been pre-assigned to known pathways or functional categories and you seek to validate only those assignments rather than discover de novo structure.
Inputs
- Normalized microbe-metabolite feature attribution score matrix (S; rows=microbes, columns=well-predicted metabolites, values ∈ [−1, 1])
- Background threshold scores derived from 100+ shuffled cross-validation iterations
- Sample phenotype labels (optional, for enrichment testing)
Outputs
- Microbial module assignments (microbe ID → module ID)
- Metabolomic module assignments (metabolite ID → module ID)
- Module-level interaction network (edges connect microbes and metabolites within modules)
- Module enrichment statistics (Wilcoxon rank-sum p-values per module, mean feature values by phenotype)
How to apply
Construct a normalized feature attribution score matrix S by: (1) dividing all significant interaction scores by the background threshold identified from shuffled data and clipping values to [−1, 1]; (2) biclustering S using consensus clustering, determining the optimal number of clusters (k*) via area-under-cumulative-distribution-function (CDF) analysis with Δk threshold of 0.025; (3) extracting the resulting microbial and metabolomic module membership from the bicluster rows and columns. The rationale is that co-clustering scores reflect shared microbe-metabolite interaction patterns; normalization by background thresholds ensures modules are robust to data-specific noise; the Δk threshold balances cluster granularity against overfitting. Validate modules by testing enrichment for phenotypic labels (e.g., IBD vs. healthy) using Wilcoxon rank-sum test on mean normalized feature values within each module.
Related tools
- MiMeNet (Neural network framework that trains models to produce feature attribution scores and orchestrates consensus clustering module construction) — https://github.com/YDaiLab/MiMeNet
- scikit-learn (Provides clustering algorithms and statistical tests (Wilcoxon rank-sum) used in module construction and enrichment validation)
- consensus clustering (Core algorithm for deriving stable, robust microbial and metabolomic modules from normalized attribution matrices)
- Seaborn (Visualization of module-level interaction networks and enrichment heatmaps)
Examples
python MiMeNet_train.py -micro data/IBD/microbiome_PRISM.csv -metab data/IBD/metabolome_PRISM.csv -micro_norm None -metab_norm CLR -net_params results/IBD/network_parameters.txt -annotation data/IBD/metabolome_annotation.csv -labels data/IBD/diagnosis_PRISM.csv -num_run_cv 10 -output IBD
Evaluation signals
- Module size distribution is reasonable (not all microbes/metabolites in one cluster, not singleton clusters); median module size >2 members suggests k* selection was effective.
- Modules show statistically significant enrichment for at least one phenotype (Wilcoxon p<0.05 for ≥1 module), indicating modules capture biological signal beyond noise.
- Normalized attribution scores within modules are higher in magnitude than between-module scores; within-module mean |score| > between-module mean |score|.
- Modules remain stable across repeated consensus clustering runs (Adjusted Rand Index >0.8 between subsampled runs) or across different k* selection thresholds (Δk ∈ [0.02, 0.03]).
- Module members share annotated metabolic pathways or known microbe-metabolite interactions, validating inferred co-clustering against external knowledge.
Limitations
- Consensus clustering assumes interaction patterns are truly modular; if microbe-metabolite interactions form a single continuum or highly interconnected network, clustering may produce artificial partitions.
- Module membership is deterministic only for the k* selected via CDF-Δk; small changes in background threshold or data composition may alter k*, affecting downstream module assignments.
- Modules are constructed from normalized attribution scores, not raw counts or absolute interaction strength; scores in [−1, 1] obscure quantitative differences in microbe-metabolite effect sizes.
- Not all metabolites may be associated with microbes; metabolites with truly microbe-independent biology will form singleton or noise-dominated modules, inflating false cluster count.
- Module enrichment tests (Wilcoxon rank-sum) assume independent samples; longitudinal or repeated-measure designs may violate assumptions and require stratified or mixed-model alternatives.
Evidence
- [results] construct a score matrix of microbe-metabolite feature attributions between the microbes and well-predicted: "using the learned network weights obtained from cross-validation training, MiMeNet constructs a score matrix of microbe-metabolite feature attributions between the microbes and well-predicted"
- [methods] normalize and clip attribution scores to construct modules: "We normalized the values in each feature attribution score matrix Si by dividing the significant threshold score identified from the background and clipped values to be between -1 and 1"
- [methods] area-under-CDF with Δk threshold for k* determination: "construct microbial and metabolite modules via consensus clustering (k* determined by area-under-CDF with Δk threshold=0.025)"
- [results] bicluster score matrix into modules: "Then MiMeNet biclusters the score matrix into microbial and metabolomic modules"
- [methods] Wilcoxon rank-sum test for module enrichment: "we determine if a module is enriched for one patient group (IBD or healthy) by comparing the average normalized feature values of the members within the module between the two groups using the IBD"
- [abstract] MiMeNet can group microbes and metabolites with similar interaction patterns: "MiMeNet can group microbes and metabolites with similar interaction patterns and functions to illuminate the underlying structure of the microbe-metabolite interaction network"
- [readme] README consensus clustering via biclustering: "This interaction score matrix is biclustered into microbe and metabolite modules, grouping sets of microbes and metabolites with similar interaction patterns."
1---2name: consensus-clustering-module-construction3description: Use when after training a neural network model on paired microbiome-metabolome data and extracting microbe-metabolite feature attribution scores for significant interactions (e.g., at the 97.5th percentile threshold).4license: CC-BY-4.05---67# Consensus-clustering module construction89## Summary1011Group microbes and metabolites with similar interaction patterns into functional modules by biclustering normalized feature attribution score matrices, using consensus clustering with data-driven k* selection. This recovers the latent structure of microbe-metabolite interaction networks and enables module-level biomarker discovery.1213## When to use1415After training a neural network model on paired microbiome-metabolome data and extracting microbe-metabolite feature attribution scores for significant interactions (e.g., at the 97.5th percentile threshold). Use this skill when you want to group co-interacting microbes and metabolites to illuminate functional relationships, especially when individual microbial or metabolite annotations are sparse or when you seek to validate predicted modules against known disease states.1617## When NOT to use1819- Attribution score matrix contains <10 significant microbe-metabolite interactions; consensus clustering requires sufficient data density to recover meaningful structure.20- Background distribution was not generated (i.e., no shuffled cross-validation performed); modules would lack statistical grounding and risk spurious clustering.21- Microbes or metabolites have already been pre-assigned to known pathways or functional categories and you seek to validate only those assignments rather than discover de novo structure.2223## Inputs2425- Normalized microbe-metabolite feature attribution score matrix (S; rows=microbes, columns=well-predicted metabolites, values ∈ [−1, 1])26- Background threshold scores derived from 100+ shuffled cross-validation iterations27- Sample phenotype labels (optional, for enrichment testing)2829## Outputs3031- Microbial module assignments (microbe ID → module ID)32- Metabolomic module assignments (metabolite ID → module ID)33- Module-level interaction network (edges connect microbes and metabolites within modules)34- Module enrichment statistics (Wilcoxon rank-sum p-values per module, mean feature values by phenotype)3536## How to apply3738Construct a normalized feature attribution score matrix S by: (1) dividing all significant interaction scores by the background threshold identified from shuffled data and clipping values to [−1, 1]; (2) biclustering S using consensus clustering, determining the optimal number of clusters (k*) via area-under-cumulative-distribution-function (CDF) analysis with Δk threshold of 0.025; (3) extracting the resulting microbial and metabolomic module membership from the bicluster rows and columns. The rationale is that co-clustering scores reflect shared microbe-metabolite interaction patterns; normalization by background thresholds ensures modules are robust to data-specific noise; the Δk threshold balances cluster granularity against overfitting. Validate modules by testing enrichment for phenotypic labels (e.g., IBD vs. healthy) using Wilcoxon rank-sum test on mean normalized feature values within each module.3940## Related tools4142- **MiMeNet** (Neural network framework that trains models to produce feature attribution scores and orchestrates consensus clustering module construction) — https://github.com/YDaiLab/MiMeNet43- **scikit-learn** (Provides clustering algorithms and statistical tests (Wilcoxon rank-sum) used in module construction and enrichment validation)44- **consensus clustering** (Core algorithm for deriving stable, robust microbial and metabolomic modules from normalized attribution matrices)45- **Seaborn** (Visualization of module-level interaction networks and enrichment heatmaps)4647## Examples4849```50python MiMeNet_train.py -micro data/IBD/microbiome_PRISM.csv -metab data/IBD/metabolome_PRISM.csv -micro_norm None -metab_norm CLR -net_params results/IBD/network_parameters.txt -annotation data/IBD/metabolome_annotation.csv -labels data/IBD/diagnosis_PRISM.csv -num_run_cv 10 -output IBD51```5253## Evaluation signals5455- Module size distribution is reasonable (not all microbes/metabolites in one cluster, not singleton clusters); median module size >2 members suggests k* selection was effective.56- Modules show statistically significant enrichment for at least one phenotype (Wilcoxon p<0.05 for ≥1 module), indicating modules capture biological signal beyond noise.57- Normalized attribution scores within modules are higher in magnitude than between-module scores; within-module mean |score| > between-module mean |score|.58- Modules remain stable across repeated consensus clustering runs (Adjusted Rand Index >0.8 between subsampled runs) or across different k* selection thresholds (Δk ∈ [0.02, 0.03]).59- Module members share annotated metabolic pathways or known microbe-metabolite interactions, validating inferred co-clustering against external knowledge.6061## Limitations6263- Consensus clustering assumes interaction patterns are truly modular; if microbe-metabolite interactions form a single continuum or highly interconnected network, clustering may produce artificial partitions.64- Module membership is deterministic only for the k* selected via CDF-Δk; small changes in background threshold or data composition may alter k*, affecting downstream module assignments.65- Modules are constructed from normalized attribution scores, not raw counts or absolute interaction strength; scores in [−1, 1] obscure quantitative differences in microbe-metabolite effect sizes.66- Not all metabolites may be associated with microbes; metabolites with truly microbe-independent biology will form singleton or noise-dominated modules, inflating false cluster count.67- Module enrichment tests (Wilcoxon rank-sum) assume independent samples; longitudinal or repeated-measure designs may violate assumptions and require stratified or mixed-model alternatives.6869## Evidence7071- [results] construct a score matrix of microbe-metabolite feature attributions between the microbes and well-predicted: "using the learned network weights obtained from cross-validation training, MiMeNet constructs a score matrix of microbe-metabolite feature attributions between the microbes and well-predicted"72- [methods] normalize and clip attribution scores to construct modules: "We normalized the values in each feature attribution score matrix Si by dividing the significant threshold score identified from the background and clipped values to be between -1 and 1"73- [methods] area-under-CDF with Δk threshold for k* determination: "construct microbial and metabolite modules via consensus clustering (k* determined by area-under-CDF with Δk threshold=0.025)"74- [results] bicluster score matrix into modules: "Then MiMeNet biclusters the score matrix into microbial and metabolomic modules"75- [methods] Wilcoxon rank-sum test for module enrichment: "we determine if a module is enriched for one patient group (IBD or healthy) by comparing the average normalized feature values of the members within the module between the two groups using the IBD"76- [abstract] MiMeNet can group microbes and metabolites with similar interaction patterns: "MiMeNet can group microbes and metabolites with similar interaction patterns and functions to illuminate the underlying structure of the microbe-metabolite interaction network"77- [readme] README consensus clustering via biclustering: "This interaction score matrix is biclustered into microbe and metabolite modules, grouping sets of microbes and metabolites with similar interaction patterns."
