neural-network-hyperparameter-tuning
Summary
Systematic selection of multilayer perceptron hyperparameters (layer size, number of layers, L2 regularization, dropout rate) optimized per dataset to maximize metabolite prediction accuracy. This skill involves empirical tuning via cross-validation to identify dataset-specific configurations that balance model complexity and generalization.
When to use
When training a multilayer perceptron to predict metabolomic features from microbiome abundances, you need dataset-specific hyperparameter configurations because prediction performance (Spearman correlation and well-predicted metabolite counts) varies substantially across microbiome environments. Use this skill when you have paired microbiome-metabolome data and must select layer architecture, regularization strength, and dropout before initiating cross-validated model evaluation.
When NOT to use
- Data has not yet been filtered to remove features present in <10% of samples; preprocessing must complete before tuning.
- You have only a single small dataset (n<50 samples) without external validation cohorts; overfitting risk is high and cross-validation estimates unreliable.
- Hyperparameters are already fixed by a prior study or pre-trained model transfer task; tuning is then redundant.
Inputs
- Comma-delimited microbial abundance table (samples × features, CLR or RA normalized, features present in ≥10% of samples)
- Comma-delimited metabolite abundance table (samples × features, CLR or RA normalized, features present in ≥10% of samples)
- Candidate hyperparameter grid or JSON specification file (net_params)
Outputs
- Optimal net_params JSON file encoding selected layer size, number of layers, L2 penalty (λ), and dropout rate
- Cross-validation performance metrics (mean Spearman correlation coefficients per metabolite, counts of well-predicted metabolites above 95th percentile threshold)
How to apply
Conduct grid search or empirical exploration across candidate hyperparameter combinations (layer sizes: 128–512; number of layers: 1–2; L2 regularization λ: 0.00001–0.005; dropout rates: 0.3–0.5) using a representative subset or the full training data with internal cross-validation (e.g., 5-fold). For each configuration, train the MLPNN with ReLU activation, ADAM optimizer, and MSE loss with early stopping (40 epochs without validation loss improvement). Select the configuration that maximizes mean Spearman correlation coefficient across folds. Document the optimal configuration in a JSON file (net_params) for reproducibility. The rationale is that different microbiome environments (gut, lung, soil) exhibit different signal-to-noise ratios and feature dimensionality: the IBD (PRISM) dataset required layer_size=512 with 1 layer and λ=0.001, while cystic fibrosis and soil required layer_size=128 with 1–2 layers and λ=0.0001–0.005, reflecting reduced metabolite complexity in smaller cohorts.
Related tools
- TensorFlow or PyTorch (Deep learning framework for implementing multilayer perceptron with ReLU activation, ADAM optimizer, MSE loss, and early stopping)
- scikit-learn (Cross-validation partitioning (KFold) and model evaluation utilities)
- MiMeNet (Reference implementation that encapsulates hyperparameter tuning via net_params JSON and -num_cv / -num_run_cv arguments) — https://github.com/YDaiLab/MiMeNet
- SciPy (Spearman correlation coefficient computation for cross-validation performance assessment)
Examples
python MiMeNet_train.py -micro data/IBD/microbiome_PRISM.csv -metab data/IBD/metabolome_PRISM.csv -micro_norm None -metab_norm CLR -net_params results/IBD/network_parameters.txt -num_run_cv 10 -num_cv 10 -output IBD
Evaluation signals
- Mean Spearman correlation coefficient on held-out test folds increases or plateaus (does not degrade) compared to default or naive hyperparameter choices
- Count of well-predicted metabolites (SCC > 95th percentile of background) is maximized; for IBD (PRISM) this should be ≥6857, for CF ≥143, for soil ≥29
- Early stopping triggers within ≤40 epochs on validation set, indicating learning curves have plateaued and overfitting is controlled
- Cross-validation performance is reproducible across multiple independent CV splits; low variance in SCC distributions across folds indicates stable hyperparameter choice
- Net_params JSON file is syntactically valid and contains all required keys (layer_size, num_layers, lambda, dropout) as JSON scalars or lists
Limitations
- Hyperparameter tuning is dataset-specific; configurations optimal for IBD gut microbiota (layer_size=512, λ=0.001) are not optimal for cystic fibrosis lung (layer_size=128, λ=0.0001) or soil (layer_size=128, λ=0.005), requiring re-tuning for each new microbiome context.
- Small microbiome cohorts (n<50) may not reliably estimate performance via k-fold cross-validation; external validation on an independent cohort is necessary to confirm generalization.
- Tuning is computationally expensive; 10 iterations of 10-fold cross-validation with background shuffling (100 models per condition) can require hours to days depending on feature dimensionality and hardware.
- The skill does not incorporate mechanistic knowledge of microbe-metabolite biology; optimal hyperparameters are purely data-driven and may not reflect underlying biological structure.
- Early stopping threshold (40 epochs without validation improvement) is fixed in the reference implementation; sensitivity to this choice is not systematically explored.
Evidence
- [other] MiMeNet (multi-layer perceptron (MLPNN) with ReLU activation, L2 regularization λ=0.001, dropout=0.5, layer size=512, 1 hidden layer) using 10 iterations of 10-fold cross-validation: "MiMeNet (multi-layer perceptron with ReLU activation, L2 regularization λ=0.001, dropout=0.5, layer size=512, 1 hidden layer)"
- [other] IBD PRISM: layer size 512, 1 layer, λ=0.001, dropout=0.5; CF: layer size 128, 1 layer, λ=0.0001, dropout=0.5; soil: layer size 128, 2 layers, λ=0.005, dropout=0.3: "optimal hyperparameters (IBD PRISM: layer size 512, 1 layer, λ=0.001, dropout=0.5; CF: layer size 128, 1 layer, λ=0.0001, dropout=0.5; soil: layer size 128, 2 layers, λ=0.005, dropout=0.3)"
- [other] training on 80% of each fold, validating on 20%, and testing on held-out 10% using mean squared error loss with L2 regularization, ReLU activation, ADAM optimizer, and early stopping at 40 iterations without validation loss improvement: "training on 80% of each fold, validating on 20%, and testing on held-out 10% using mean squared error loss with L2 regularization, ReLU activation, ADAM optimizer, and early stopping at 40 iterations"
- [readme] net_params JSON file containing neural network number of layers, layer size, L2 penalty, and dropout rate: "JSON file containing neural network number of layers, layer size, L2 penalty, and dropout rate"
- [other] different dataset was taken from a study that collected 172 lung sputum samples from patients with cystic fibrosis exhibiting mean Spearman correlations of 0.457, and soil dataset exhibited 0.264, contrasting with IBD 0.309: "mean Spearman correlation coefficients of 0.309, 0.457, and 0.264 for IBD (PRISM), cystic fibrosis, and soil datasets respectively"
1---2name: neural-network-hyperparameter-tuning-23description: Use when when training a multilayer perceptron to predict metabolomic features from microbiome abundances, you need dataset-specific hyperparameter configurations because prediction performance (Spearman correlation and well-predicted metabolite counts) varies substantially across microbiome.4license: CC-BY-4.05---67# neural-network-hyperparameter-tuning89## Summary1011Systematic selection of multilayer perceptron hyperparameters (layer size, number of layers, L2 regularization, dropout rate) optimized per dataset to maximize metabolite prediction accuracy. This skill involves empirical tuning via cross-validation to identify dataset-specific configurations that balance model complexity and generalization.1213## When to use1415When training a multilayer perceptron to predict metabolomic features from microbiome abundances, you need dataset-specific hyperparameter configurations because prediction performance (Spearman correlation and well-predicted metabolite counts) varies substantially across microbiome environments. Use this skill when you have paired microbiome-metabolome data and must select layer architecture, regularization strength, and dropout before initiating cross-validated model evaluation.1617## When NOT to use1819- Data has not yet been filtered to remove features present in <10% of samples; preprocessing must complete before tuning.20- You have only a single small dataset (n<50 samples) without external validation cohorts; overfitting risk is high and cross-validation estimates unreliable.21- Hyperparameters are already fixed by a prior study or pre-trained model transfer task; tuning is then redundant.2223## Inputs2425- Comma-delimited microbial abundance table (samples × features, CLR or RA normalized, features present in ≥10% of samples)26- Comma-delimited metabolite abundance table (samples × features, CLR or RA normalized, features present in ≥10% of samples)27- Candidate hyperparameter grid or JSON specification file (net_params)2829## Outputs3031- Optimal net_params JSON file encoding selected layer size, number of layers, L2 penalty (λ), and dropout rate32- Cross-validation performance metrics (mean Spearman correlation coefficients per metabolite, counts of well-predicted metabolites above 95th percentile threshold)3334## How to apply3536Conduct grid search or empirical exploration across candidate hyperparameter combinations (layer sizes: 128–512; number of layers: 1–2; L2 regularization λ: 0.00001–0.005; dropout rates: 0.3–0.5) using a representative subset or the full training data with internal cross-validation (e.g., 5-fold). For each configuration, train the MLPNN with ReLU activation, ADAM optimizer, and MSE loss with early stopping (40 epochs without validation loss improvement). Select the configuration that maximizes mean Spearman correlation coefficient across folds. Document the optimal configuration in a JSON file (net_params) for reproducibility. The rationale is that different microbiome environments (gut, lung, soil) exhibit different signal-to-noise ratios and feature dimensionality: the IBD (PRISM) dataset required layer_size=512 with 1 layer and λ=0.001, while cystic fibrosis and soil required layer_size=128 with 1–2 layers and λ=0.0001–0.005, reflecting reduced metabolite complexity in smaller cohorts.3738## Related tools3940- **TensorFlow or PyTorch** (Deep learning framework for implementing multilayer perceptron with ReLU activation, ADAM optimizer, MSE loss, and early stopping)41- **scikit-learn** (Cross-validation partitioning (KFold) and model evaluation utilities)42- **MiMeNet** (Reference implementation that encapsulates hyperparameter tuning via net_params JSON and -num_cv / -num_run_cv arguments) — https://github.com/YDaiLab/MiMeNet43- **SciPy** (Spearman correlation coefficient computation for cross-validation performance assessment)4445## Examples4647```48python MiMeNet_train.py -micro data/IBD/microbiome_PRISM.csv -metab data/IBD/metabolome_PRISM.csv -micro_norm None -metab_norm CLR -net_params results/IBD/network_parameters.txt -num_run_cv 10 -num_cv 10 -output IBD49```5051## Evaluation signals5253- Mean Spearman correlation coefficient on held-out test folds increases or plateaus (does not degrade) compared to default or naive hyperparameter choices54- Count of well-predicted metabolites (SCC > 95th percentile of background) is maximized; for IBD (PRISM) this should be ≥6857, for CF ≥143, for soil ≥2955- Early stopping triggers within ≤40 epochs on validation set, indicating learning curves have plateaued and overfitting is controlled56- Cross-validation performance is reproducible across multiple independent CV splits; low variance in SCC distributions across folds indicates stable hyperparameter choice57- Net_params JSON file is syntactically valid and contains all required keys (layer_size, num_layers, lambda, dropout) as JSON scalars or lists5859## Limitations6061- Hyperparameter tuning is dataset-specific; configurations optimal for IBD gut microbiota (layer_size=512, λ=0.001) are not optimal for cystic fibrosis lung (layer_size=128, λ=0.0001) or soil (layer_size=128, λ=0.005), requiring re-tuning for each new microbiome context.62- Small microbiome cohorts (n<50) may not reliably estimate performance via k-fold cross-validation; external validation on an independent cohort is necessary to confirm generalization.63- Tuning is computationally expensive; 10 iterations of 10-fold cross-validation with background shuffling (100 models per condition) can require hours to days depending on feature dimensionality and hardware.64- The skill does not incorporate mechanistic knowledge of microbe-metabolite biology; optimal hyperparameters are purely data-driven and may not reflect underlying biological structure.65- Early stopping threshold (40 epochs without validation improvement) is fixed in the reference implementation; sensitivity to this choice is not systematically explored.6667## Evidence6869- [other] MiMeNet (multi-layer perceptron (MLPNN) with ReLU activation, L2 regularization λ=0.001, dropout=0.5, layer size=512, 1 hidden layer) using 10 iterations of 10-fold cross-validation: "MiMeNet (multi-layer perceptron with ReLU activation, L2 regularization λ=0.001, dropout=0.5, layer size=512, 1 hidden layer)"70- [other] IBD PRISM: layer size 512, 1 layer, λ=0.001, dropout=0.5; CF: layer size 128, 1 layer, λ=0.0001, dropout=0.5; soil: layer size 128, 2 layers, λ=0.005, dropout=0.3: "optimal hyperparameters (IBD PRISM: layer size 512, 1 layer, λ=0.001, dropout=0.5; CF: layer size 128, 1 layer, λ=0.0001, dropout=0.5; soil: layer size 128, 2 layers, λ=0.005, dropout=0.3)"71- [other] training on 80% of each fold, validating on 20%, and testing on held-out 10% using mean squared error loss with L2 regularization, ReLU activation, ADAM optimizer, and early stopping at 40 iterations without validation loss improvement: "training on 80% of each fold, validating on 20%, and testing on held-out 10% using mean squared error loss with L2 regularization, ReLU activation, ADAM optimizer, and early stopping at 40 iterations"72- [readme] net_params JSON file containing neural network number of layers, layer size, L2 penalty, and dropout rate: "JSON file containing neural network number of layers, layer size, L2 penalty, and dropout rate"73- [other] different dataset was taken from a study that collected 172 lung sputum samples from patients with cystic fibrosis exhibiting mean Spearman correlations of 0.457, and soil dataset exhibited 0.264, contrasting with IBD 0.309: "mean Spearman correlation coefficients of 0.309, 0.457, and 0.264 for IBD (PRISM), cystic fibrosis, and soil datasets respectively"
HolobiomicsLab 