Spearman Correlation Coefficient Calculation across Cross-Validation Folds
Summary
Compute Spearman correlation coefficients (SCC) between predicted and observed metabolite abundances across multiple cross-validation iterations and folds to assess prediction accuracy and identify well-predicted metabolites in microbiome-metabolome modeling. This skill quantifies how well a neural network model captures the rank-order relationship between predicted and observed metabolomic profiles.
When to use
You have trained predictive models (e.g., MiMeNet neural networks) on microbiome-metabolome paired data using k-fold cross-validation, held out test folds for each fold-iteration, and generated continuous abundance predictions for metabolites. SCC is appropriate when your outcome and prediction are both continuous (metabolite relative abundances) and you want to measure monotonic association regardless of scaling. Apply this skill to evaluate prediction performance per metabolite and rank metabolites by predictability for downstream module construction or biomarker discovery.
When NOT to use
- Input abundances are binary or categorical (e.g., presence/absence) — use appropriate classification metrics (AUROC, F1) instead.
- You have only a single train-test split without cross-validation — use Spearman SCC directly on that split, but do not aggregate across folds.
- Predicted abundances are not continuous (e.g., class probabilities or discrete abundance bins) — validate that both vectors are continuous before applying Spearman.
Inputs
- Predicted metabolite abundance matrix (samples × metabolites) from trained neural network model on held-out test fold
- Observed metabolite abundance matrix (samples × metabolites) from the same test fold
- Cross-validation iteration and fold identifiers to track which data belong to which split
Outputs
- Per-metabolite Spearman correlation coefficient (SCC) for each fold-iteration (matrix: metabolites × 100 fold-iterations)
- Mean SCC per metabolite (vector: metabolites, with one value per metabolite)
- Count of well-predicted metabolites (metabolites with SCC > 95th percentile threshold)
- Background distribution of SCCs (generated on shuffled data for threshold determination)
How to apply
For each of the k×m test folds (k folds per iteration, m iterations), compute the Spearman rank correlation coefficient between the vector of predicted metabolite abundances and the vector of observed (ground-truth) abundances for each metabolite. Aggregate SCCs across all folds (typically 10 iterations × 10 folds = 100 model evaluations) to generate a mean SCC per metabolite. Use the resulting per-metabolite SCC distribution to identify 'well-predicted' metabolites—those with SCC values above an empirically determined threshold (commonly the 95th percentile of a background distribution generated by shuffling sample labels during cross-validation). This ranked list of metabolites by SCC enables filtering of high-confidence predictions and identification of microbiome-associated metabolites for mechanistic analysis.
Related tools
- MiMeNet (Neural network framework that generates metabolite abundance predictions on held-out test folds, outputs of which feed into SCC calculation) — https://github.com/YDaiLab/MiMeNet
- scipy.stats.spearmanr (Python) (Computes Spearman rank correlation coefficient between predicted and observed vectors for each metabolite)
- scikit-learn (Python) (Provides cross-validation framework (KFold, cross_validate) to partition data and iterate folds)
Examples
from scipy.stats import spearmanr; import numpy as np; predicted = model.predict(X_test); scc_per_metabolite = [spearmanr(predicted[:, i], y_test[:, i])[0] for i in range(predicted.shape[1])]; mean_scc = np.mean(scc_per_metabolite); well_predicted_count = np.sum(np.array(scc_per_metabolite) > np.percentile(background_scc, 95))
Evaluation signals
- SCC values must lie in the range [−1, +1]; metabolites with SCC > 0.5 are considered reasonably well-predicted (article baseline: SCC mean of 0.309 for IBD PRISM dataset)
- Well-predicted metabolite count must be significantly higher than expected by chance; compare observed count to count from 95th percentile of shuffled (background) SCC distribution
- Mean SCC across all 100 fold-iterations must be stable (low variance) and reproducible across independent runs; high variance indicates unstable model or noisy data
- SCC per metabolite should correlate with downstream module membership — metabolites in the same biclustered module should have similar SCC rankings and interaction profiles
- Comparison between two training regimes (e.g., all metabolites vs. annotated-only) should show a measurable delta in mean SCC (e.g., difference of 0.05 or greater with statistical significance p < 0.001)
Limitations
- Not all metabolites may be associated with microbes; metabolites with weak microbe-metabolite associations will have lower prediction correlations and reduce overall mean SCC across the metabolite set.
- Spearman SCC depends on sample size and rank variability; small test folds (n < 20) may yield unstable SCC estimates per metabolite, requiring larger k or m.
- Background distribution method (shuffling labels) assumes that shuffled data have no microbe-metabolome relationship; if some signal persists despite shuffling, the 95th percentile threshold may be too lenient.
- Cross-validation strategy (e.g., stratified vs. random, number of folds) can affect SCC estimates; imbalanced or small datasets may require careful fold construction to ensure representative test sets.
Evidence
- [other] calculate Spearman correlation coefficient (SCC) between predicted and observed abundance for each annotated metabolite across all 100 model runs (10 iterations × 10 folds): "calculate Spearman correlation coefficient (SCC) between predicted and observed abundance for each annotated metabolite across all 100 model runs (10 iterations × 10 folds)"
- [results] the predictive performance of a model is measured by the average Spearman correlation coefficients (SCCs) between the predicted and the observed abundances: "the predictive performance of a model is measured by the average Spearman correlation coefficients (SCCs) between the predicted and the observed abundances"
- [methods] We then defined a metabolite as well-predicted if its SCC is above the 95th percentile of the background correlations: "We then defined a metabolite as well-predicted if its SCC is above the 95th percentile of the background correlations"
- [readme] MiMeNet generates a background of SCC values using a similar approach as in Cross-Validated Evaluation. However, to generate the background distribution of SCCs, the samples are randomly shuffled for each cross-validated iteration.: "MiMeNet generates a background of SCC values using a similar approach as in Cross-Validated Evaluation. However, to generate the background distribution of SCCs, the samples are randomly shuffled for"
- [other] Training MiMeNet on all metabolites improved mean Spearman correlation coefficients for annotated metabolites from 0.259 to 0.309 (P < 10−47), with well-predicted metabolites increasing from 333 to 366 out of 466 annotated metabolites: "Training MiMeNet on all metabolites improved mean Spearman correlation coefficients for annotated metabolites from 0.259 to 0.309 (P < 10−47), with well-predicted metabolites increasing from 333 to"
1---2name: spearman-correlation-coefficient-calculation-cross-valid-23description: Use when you have trained predictive models (e.g., MiMeNet neural networks) on microbiome-metabolome paired data using k-fold cross-validation, held out test folds for each fold-iteration, and generated continuous abundance predictions for metabolites.4license: CC-BY-4.05---67# Spearman Correlation Coefficient Calculation across Cross-Validation Folds89## Summary1011Compute Spearman correlation coefficients (SCC) between predicted and observed metabolite abundances across multiple cross-validation iterations and folds to assess prediction accuracy and identify well-predicted metabolites in microbiome-metabolome modeling. This skill quantifies how well a neural network model captures the rank-order relationship between predicted and observed metabolomic profiles.1213## When to use1415You have trained predictive models (e.g., MiMeNet neural networks) on microbiome-metabolome paired data using k-fold cross-validation, held out test folds for each fold-iteration, and generated continuous abundance predictions for metabolites. SCC is appropriate when your outcome and prediction are both continuous (metabolite relative abundances) and you want to measure monotonic association regardless of scaling. Apply this skill to evaluate prediction performance per metabolite and rank metabolites by predictability for downstream module construction or biomarker discovery.1617## When NOT to use1819- Input abundances are binary or categorical (e.g., presence/absence) — use appropriate classification metrics (AUROC, F1) instead.20- You have only a single train-test split without cross-validation — use Spearman SCC directly on that split, but do not aggregate across folds.21- Predicted abundances are not continuous (e.g., class probabilities or discrete abundance bins) — validate that both vectors are continuous before applying Spearman.2223## Inputs2425- Predicted metabolite abundance matrix (samples × metabolites) from trained neural network model on held-out test fold26- Observed metabolite abundance matrix (samples × metabolites) from the same test fold27- Cross-validation iteration and fold identifiers to track which data belong to which split2829## Outputs3031- Per-metabolite Spearman correlation coefficient (SCC) for each fold-iteration (matrix: metabolites × 100 fold-iterations)32- Mean SCC per metabolite (vector: metabolites, with one value per metabolite)33- Count of well-predicted metabolites (metabolites with SCC > 95th percentile threshold)34- Background distribution of SCCs (generated on shuffled data for threshold determination)3536## How to apply3738For each of the k×m test folds (k folds per iteration, m iterations), compute the Spearman rank correlation coefficient between the vector of predicted metabolite abundances and the vector of observed (ground-truth) abundances for each metabolite. Aggregate SCCs across all folds (typically 10 iterations × 10 folds = 100 model evaluations) to generate a mean SCC per metabolite. Use the resulting per-metabolite SCC distribution to identify 'well-predicted' metabolites—those with SCC values above an empirically determined threshold (commonly the 95th percentile of a background distribution generated by shuffling sample labels during cross-validation). This ranked list of metabolites by SCC enables filtering of high-confidence predictions and identification of microbiome-associated metabolites for mechanistic analysis.3940## Related tools4142- **MiMeNet** (Neural network framework that generates metabolite abundance predictions on held-out test folds, outputs of which feed into SCC calculation) — https://github.com/YDaiLab/MiMeNet43- **scipy.stats.spearmanr (Python)** (Computes Spearman rank correlation coefficient between predicted and observed vectors for each metabolite)44- **scikit-learn (Python)** (Provides cross-validation framework (KFold, cross_validate) to partition data and iterate folds)4546## Examples4748```49from scipy.stats import spearmanr; import numpy as np; predicted = model.predict(X_test); scc_per_metabolite = [spearmanr(predicted[:, i], y_test[:, i])[0] for i in range(predicted.shape[1])]; mean_scc = np.mean(scc_per_metabolite); well_predicted_count = np.sum(np.array(scc_per_metabolite) > np.percentile(background_scc, 95))50```5152## Evaluation signals5354- SCC values must lie in the range [−1, +1]; metabolites with SCC > 0.5 are considered reasonably well-predicted (article baseline: SCC mean of 0.309 for IBD PRISM dataset)55- Well-predicted metabolite count must be significantly higher than expected by chance; compare observed count to count from 95th percentile of shuffled (background) SCC distribution56- Mean SCC across all 100 fold-iterations must be stable (low variance) and reproducible across independent runs; high variance indicates unstable model or noisy data57- SCC per metabolite should correlate with downstream module membership — metabolites in the same biclustered module should have similar SCC rankings and interaction profiles58- Comparison between two training regimes (e.g., all metabolites vs. annotated-only) should show a measurable delta in mean SCC (e.g., difference of 0.05 or greater with statistical significance p < 0.001)5960## Limitations6162- Not all metabolites may be associated with microbes; metabolites with weak microbe-metabolite associations will have lower prediction correlations and reduce overall mean SCC across the metabolite set.63- Spearman SCC depends on sample size and rank variability; small test folds (n < 20) may yield unstable SCC estimates per metabolite, requiring larger k or m.64- Background distribution method (shuffling labels) assumes that shuffled data have no microbe-metabolome relationship; if some signal persists despite shuffling, the 95th percentile threshold may be too lenient.65- Cross-validation strategy (e.g., stratified vs. random, number of folds) can affect SCC estimates; imbalanced or small datasets may require careful fold construction to ensure representative test sets.6667## Evidence6869- [other] calculate Spearman correlation coefficient (SCC) between predicted and observed abundance for each annotated metabolite across all 100 model runs (10 iterations × 10 folds): "calculate Spearman correlation coefficient (SCC) between predicted and observed abundance for each annotated metabolite across all 100 model runs (10 iterations × 10 folds)"70- [results] the predictive performance of a model is measured by the average Spearman correlation coefficients (SCCs) between the predicted and the observed abundances: "the predictive performance of a model is measured by the average Spearman correlation coefficients (SCCs) between the predicted and the observed abundances"71- [methods] We then defined a metabolite as well-predicted if its SCC is above the 95th percentile of the background correlations: "We then defined a metabolite as well-predicted if its SCC is above the 95th percentile of the background correlations"72- [readme] MiMeNet generates a background of SCC values using a similar approach as in Cross-Validated Evaluation. However, to generate the background distribution of SCCs, the samples are randomly shuffled for each cross-validated iteration.: "MiMeNet generates a background of SCC values using a similar approach as in Cross-Validated Evaluation. However, to generate the background distribution of SCCs, the samples are randomly shuffled for"73- [other] Training MiMeNet on all metabolites improved mean Spearman correlation coefficients for annotated metabolites from 0.259 to 0.309 (P < 10−47), with well-predicted metabolites increasing from 333 to 366 out of 466 annotated metabolites: "Training MiMeNet on all metabolites improved mean Spearman correlation coefficients for annotated metabolites from 0.259 to 0.309 (P < 10−47), with well-predicted metabolites increasing from 333 to"
HolobiomicsLab 