Machine-Learning Hyperparameter Extraction
License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution.
Summary
Extract and validate optimized hyperparameter values (e.g., nIter, method) for trained machine-learning classifiers from cross-validation output. This skill identifies which hyperparameter combinations were selected during model tuning and documents them for reproducibility and model retraining.
When to use
You have completed cross-validation tuning of one or more machine-learning models (e.g., AdaBoost, SVM, Random Forest) on a development dataset using the caret package and need to identify which specific hyperparameter values were selected as optimal. This is necessary before training a final classifier on the full development set or before applying the model to held-out test data.
When NOT to use
- You are using pre-trained models from external sources without access to their cross-validation history or prediction data frames.
- Hyperparameter tuning was not performed; the model uses default hyperparameters and no runCrossValidation() output is available.
- The evaluation measures output is unavailable or does not align with the pred data frame (mismatched cross-validation runs).
Inputs
- List object returned by runCrossValidation() containing trained models
- Prediction data frame from models$[AlgorithmName]$pred
- Evaluation measures data frame from getEvaluationMeasures()
- Cross-validation parameters (k-fold number, repetition number, metric set)
Outputs
- Extracted hyperparameter combinations (e.g., nIter, method values)
- Identified optimized hyperparameter set for best-performing model configuration
- Validated correspondence between hyperparameters and evaluation metrics
How to apply
After running runCrossValidation() with specified k-fold and repetition parameters (e.g., k=5, repNum=10) and a selected metric set (e.g., metricSet=c('M4','M7','M11')), access the prediction data frame returned by the trained model object (e.g., models$AdaBoost_M11$pred). Extract the unique hyperparameter columns (nIter, method, etc.) from this pred data frame to identify all candidate hyperparameter combinations tested. Cross-reference the extracted combinations against the evaluation measures output (generated by getEvaluationMeasures()) to confirm which hyperparameter set corresponds to the best-performing configuration according to your chosen evaluation metric. Document the optimized values for use in the subsequent trainClassifier() call on the full development set.
Related tools
- caret (R package used to implement machine-learning algorithms and manage cross-validation hyperparameter tuning)
- MetaClean (Peak-quality classifier package providing wrapper functions runCrossValidation(), getEvaluationMeasures(), and trainClassifier() that manage hyperparameter tuning and extraction workflow) — https://github.com/KelseyChetnik/MetaClean
- R (Statistical computing environment in which hyperparameter extraction, data frame manipulation, and validation are performed)
Examples
models <- runCrossValidation(trainData = pqm_development, k = 5, repNum = 10, rand.seed = 512, models = 'all', metricSet = c('M4','M7','M11')); hyperparams <- unique(models$AdaBoost_M11$pred[, c('nIter', 'method')]); eval_measures <- getEvaluationMeasures(models); best_idx <- which.max(eval_measures$Accuracy); optimal_hyperparams <- hyperparams[best_idx, ]
Evaluation signals
- The extracted hyperparameter columns (nIter, method) are present and non-null in the pred data frame.
- The unique hyperparameter combinations extracted match the set of combinations that were iterated during cross-validation (no spurious or missing values).
- The optimized hyperparameter set identified can be cross-referenced to the row(s) in the evaluation measures output with the best (highest or lowest, depending on metric direction) performance score.
- The hyperparameter values are numeric or character strings matching the data types expected by trainClassifier() (e.g., nIter is an integer, method is a character string such as 'Adaboost.M1').
- When the optimized hyperparameters are passed to trainClassifier() on the full development set, model training succeeds without parameter-type or range errors.
Limitations
- The article does not provide the hyperparameter values in the reported text; extraction relies entirely on programmatic access to the pred data frame, which is not published in the paper.
- Cross-validation outputs are sensitive to random seed and sampling strategy; reproducibility requires recording the exact runCrossValidation() parameters (k, repNum, rand.seed) used during tuning.
- If multiple hyperparameter combinations achieve similar evaluation scores, the extraction method does not disambiguate; practitioners must apply domain knowledge or additional criteria to select among tied candidates.
- The skill applies to MetaClean's wrapper functions and caret-compatible algorithms; extraction workflows may differ for other machine-learning frameworks or packages.
Evidence
- [other] Load the trained models list returned by runCrossValidation() with parameters k=5, repNum=10, rand.seed=512, models='all', metricSet=c('M4','M7','M11') on pqm_development (500 peaks, 89 samples). 2. Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame.: "Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame."
- [other] Identify and record the optimized hyperparameters determined during cross-validation (nIter=150, method='Adaboost.M1' per the example). 5. Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output.: "Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output."
- [methods] The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms.: "The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms."
- [methods] We use the getEvaluationMeasures function to do this.: "We use the getEvaluationMeasures function to do this."
- [methods] metaclean_model <- trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters): "trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters)"
1---2name: machine-learning-hyperparameter-extraction3description: Use when you have completed cross-validation tuning of one or more machine-learning models (e.g., AdaBoost, SVM, Random Forest) on a development dataset using the caret package and need to identify which specific hyperparameter values were selected as optimal.4license: CC-BY-4.05---67# Machine-Learning Hyperparameter Extraction89> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->10## Summary1112Extract and validate optimized hyperparameter values (e.g., nIter, method) for trained machine-learning classifiers from cross-validation output. This skill identifies which hyperparameter combinations were selected during model tuning and documents them for reproducibility and model retraining.1314## When to use1516You have completed cross-validation tuning of one or more machine-learning models (e.g., AdaBoost, SVM, Random Forest) on a development dataset using the caret package and need to identify which specific hyperparameter values were selected as optimal. This is necessary before training a final classifier on the full development set or before applying the model to held-out test data.1718## When NOT to use1920- You are using pre-trained models from external sources without access to their cross-validation history or prediction data frames.21- Hyperparameter tuning was not performed; the model uses default hyperparameters and no runCrossValidation() output is available.22- The evaluation measures output is unavailable or does not align with the pred data frame (mismatched cross-validation runs).2324## Inputs2526- List object returned by runCrossValidation() containing trained models27- Prediction data frame from models$[AlgorithmName]$pred28- Evaluation measures data frame from getEvaluationMeasures()29- Cross-validation parameters (k-fold number, repetition number, metric set)3031## Outputs3233- Extracted hyperparameter combinations (e.g., nIter, method values)34- Identified optimized hyperparameter set for best-performing model configuration35- Validated correspondence between hyperparameters and evaluation metrics3637## How to apply3839After running runCrossValidation() with specified k-fold and repetition parameters (e.g., k=5, repNum=10) and a selected metric set (e.g., metricSet=c('M4','M7','M11')), access the prediction data frame returned by the trained model object (e.g., models$AdaBoost_M11$pred). Extract the unique hyperparameter columns (nIter, method, etc.) from this pred data frame to identify all candidate hyperparameter combinations tested. Cross-reference the extracted combinations against the evaluation measures output (generated by getEvaluationMeasures()) to confirm which hyperparameter set corresponds to the best-performing configuration according to your chosen evaluation metric. Document the optimized values for use in the subsequent trainClassifier() call on the full development set.4041## Related tools4243- **caret** (R package used to implement machine-learning algorithms and manage cross-validation hyperparameter tuning)44- **MetaClean** (Peak-quality classifier package providing wrapper functions runCrossValidation(), getEvaluationMeasures(), and trainClassifier() that manage hyperparameter tuning and extraction workflow) — https://github.com/KelseyChetnik/MetaClean45- **R** (Statistical computing environment in which hyperparameter extraction, data frame manipulation, and validation are performed)4647## Examples4849```50models <- runCrossValidation(trainData = pqm_development, k = 5, repNum = 10, rand.seed = 512, models = 'all', metricSet = c('M4','M7','M11')); hyperparams <- unique(models$AdaBoost_M11$pred[, c('nIter', 'method')]); eval_measures <- getEvaluationMeasures(models); best_idx <- which.max(eval_measures$Accuracy); optimal_hyperparams <- hyperparams[best_idx, ]51```5253## Evaluation signals5455- The extracted hyperparameter columns (nIter, method) are present and non-null in the pred data frame.56- The unique hyperparameter combinations extracted match the set of combinations that were iterated during cross-validation (no spurious or missing values).57- The optimized hyperparameter set identified can be cross-referenced to the row(s) in the evaluation measures output with the best (highest or lowest, depending on metric direction) performance score.58- The hyperparameter values are numeric or character strings matching the data types expected by trainClassifier() (e.g., nIter is an integer, method is a character string such as 'Adaboost.M1').59- When the optimized hyperparameters are passed to trainClassifier() on the full development set, model training succeeds without parameter-type or range errors.6061## Limitations6263- The article does not provide the hyperparameter values in the reported text; extraction relies entirely on programmatic access to the pred data frame, which is not published in the paper.64- Cross-validation outputs are sensitive to random seed and sampling strategy; reproducibility requires recording the exact runCrossValidation() parameters (k, repNum, rand.seed) used during tuning.65- If multiple hyperparameter combinations achieve similar evaluation scores, the extraction method does not disambiguate; practitioners must apply domain knowledge or additional criteria to select among tied candidates.66- The skill applies to MetaClean's wrapper functions and caret-compatible algorithms; extraction workflows may differ for other machine-learning frameworks or packages.6768## Evidence6970- [other] Load the trained models list returned by runCrossValidation() with parameters k=5, repNum=10, rand.seed=512, models='all', metricSet=c('M4','M7','M11') on pqm_development (500 peaks, 89 samples). 2. Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame.: "Access the prediction data frame from models$AdaBoost_M11$pred. 3. Extract the unique hyperparameter combinations (nIter and method columns) from the pred data frame."71- [other] Identify and record the optimized hyperparameters determined during cross-validation (nIter=150, method='Adaboost.M1' per the example). 5. Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output.: "Validate that these hyperparameters correspond to the best-performing AdaBoost configuration by cross-referencing with evaluation measures output."72- [methods] The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms.: "The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms."73- [methods] We use the getEvaluationMeasures function to do this.: "We use the getEvaluationMeasures function to do this."74- [methods] metaclean_model <- trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters): "trainClassifier(trainData = pqm_development, model = "AdaBoost", metricSet = "M11", hyperparameters = hyperparameters)"
HolobiomicsLab