Comparative Classifier Performance Assessment
License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution.
Summary
Systematically train and rank multiple machine learning algorithms on the same peak-quality assessment task using k-fold cross-validation with statistical ranking plots to identify the best-performing classifier. This skill applies when selecting among competing algorithms for peak quality classification in LC-MS metabolomics data.
When to use
You have a labeled peak quality dataset (development set with ground-truth pass/fail labels), a defined set of peak-quality metrics (e.g., M11 metric set), and need to determine which of the 8–9 available machine learning algorithms (DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM_Linear, AdaBoost, NeuralNetwork, ModelAveragedNeuralNetwork) will perform best before deploying a final classifier on test data. Use this skill when model selection must be reproducible across multiple repetitions (e.g., 10 repetitions × 5 folds) and when statistical ranking rather than a single run is required.
When NOT to use
- Input data lacks ground-truth labels — runCrossValidation requires labeled training data, not unlabeled peak sets.
- You already have a pre-trained classifier validated on independent test data — do not re-rank algorithms; proceed to final model training instead.
- Sample size is very small (e.g., <50 labeled peaks total) — cross-validation with k=5 and repNum=10 may yield unreliable rankings due to high variance.
Inputs
- pqm_development: labeled peak quality metrics matrix with ground-truth pass/fail labels
- metric set specification (e.g., 'M11')
- random seed (integer, for reproducibility)
- cross-validation parameters (k, repNum)
Outputs
- cross-validation fold results: trained models and hyperparameters for each algorithm and fold
- evaluation measures table: Pass_FScore, Fail_FScore, Accuracy, Pass_Precision, Pass_Recall, Fail_Precision, Fail_Recall per model and round
- bar plots: visual comparison of mean scores across algorithms
- CD (Friedman-Demšar) plots and rank matrices: statistical ranking of algorithms
- best-performing algorithm identifier with mean scores
How to apply
Execute runCrossValidation on the labeled peak quality metrics matrix (pqm_development) with specified parameters: k=5 folds, repNum=10 repetitions, a fixed random seed (e.g., 453), your chosen metric set (e.g., 'M11'), and the full subset of algorithms to compare. Apply getEvaluationMeasures to extract Pass_FScore, Fail_FScore, Accuracy, and other measures across all cross-validation rounds. Visualize results using getBarPlots to compare mean performance scores across models, then generate CD (Friedman-Demšar) plots with getCDPlots (compareBest=FALSE) to obtain statistical rankings that account for multiple comparisons. Rank models by mean evaluation scores and statistical significance; the algorithm with the highest mean score and lowest critical distance is the best performer for your metric set.
Related tools
- MetaClean (R package providing runCrossValidation, getEvaluationMeasures, getBarPlots, and getCDPlots functions for multi-algorithm comparison and statistical ranking) — https://github.com/KelseyChetnik/MetaClean
- caret (underlying machine learning framework in R used by MetaClean to implement all 8 classification algorithms)
- MetaCleanData (R package providing example development and test peak-quality metrics matrices for reproducible cross-validation studies) — https://github.com/KelseyChetnik/MetaCleanData
- R (runtime environment and language for executing MetaClean functions)
Examples
runCrossValidation(pqm_development, k=5, repNum=10, rand.seed=453, algorithms=c('DecisionTree','LogisticRegression','NaiveBayes','RandomForest','SVM_Linear','AdaBoost','NeuralNetwork','ModelAveragedNeuralNetwork'), metricSet='M11')
Evaluation signals
- Cross-validation completes without errors for all k·repNum fold splits and all requested algorithms; no missing hyperparameter sets returned.
- Evaluation measures are computed for all folds and all algorithms; no NaN or infinite values in Pass_FScore, Fail_FScore, or Accuracy columns.
- Bar plots show non-overlapping or clearly distinct mean scores across at least 2–3 top-ranked algorithms, indicating discriminative ranking.
- CD plot rankings are consistent with mean scores: algorithm with highest mean Pass_FScore appears at lowest critical distance rank.
- Friedman statistical test p-value (if reported) is <0.05, indicating significant differences among algorithms; if p≥0.05, no algorithm is truly superior within the metric set.
Limitations
- Results are specific to the chosen metric set (e.g., M11); algorithm rankings may differ for other metric sets (e.g., M1–M8 or custom subsets).
- Cross-validation performance does not guarantee generalization to truly independent test data; final validation on held-out pqm_test is required before deployment.
- MetaClean uses only 8 fixed algorithms (DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM_Linear, AdaBoost, NeuralNetwork, ModelAveragedNeuralNetwork); custom or ensemble algorithms not in this set cannot be ranked using runCrossValidation.
- Hyperparameter tuning is performed by caret's internal defaults; systematic hyperparameter optimization is not described in the workflow and may affect algorithm rankings if different hyperparameters are chosen.
Evidence
- [other] 5-fold cross-validation with 10 repetitions on M11 metric set using random seed 453: "running 5-fold cross-validation with 10 repetitions on the M11 metric set using MetaCleanData with random seed 453"
- [methods] runCrossValidation function wraps cross-validation training of user-selected algorithm subset: "The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms."
- [methods] getEvaluationMeasures extracts performance metrics per model across CV rounds: "We use the getEvaluationMeasures function to do this."
- [methods] getCDPlots generates statistical CD plots and rank matrices for algorithm comparison: "MetaClean also provides a wrapper function to easily generate CD plots of the evaluation measures for each model. This is getCDPlots"
- [other] AdaBoost identified as best algorithm by comparing mean scores and statistical rankings: "Identify AdaBoost as the best-performing algorithm by comparing mean evaluation scores and statistical rankings across all models."
- [readme] 9 diverse machine learning algorithms available for training and comparison: "MetaClean utilizes 12 peak-quality metrics and 9 diverse machine learning algorithms to build a classifier to detect and flag low-quality peaks in untargeted metabolomics data."
1---2name: comparative-classifier-performance-assessment-23description: Use when you have a labeled peak quality dataset (development set with ground-truth pass/fail labels), a defined set of peak-quality metrics (e.4license: CC-BY-4.05---67# Comparative Classifier Performance Assessment89> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->10## Summary1112Systematically train and rank multiple machine learning algorithms on the same peak-quality assessment task using k-fold cross-validation with statistical ranking plots to identify the best-performing classifier. This skill applies when selecting among competing algorithms for peak quality classification in LC-MS metabolomics data.1314## When to use1516You have a labeled peak quality dataset (development set with ground-truth pass/fail labels), a defined set of peak-quality metrics (e.g., M11 metric set), and need to determine which of the 8–9 available machine learning algorithms (DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM_Linear, AdaBoost, NeuralNetwork, ModelAveragedNeuralNetwork) will perform best before deploying a final classifier on test data. Use this skill when model selection must be reproducible across multiple repetitions (e.g., 10 repetitions × 5 folds) and when statistical ranking rather than a single run is required.1718## When NOT to use1920- Input data lacks ground-truth labels — runCrossValidation requires labeled training data, not unlabeled peak sets.21- You already have a pre-trained classifier validated on independent test data — do not re-rank algorithms; proceed to final model training instead.22- Sample size is very small (e.g., <50 labeled peaks total) — cross-validation with k=5 and repNum=10 may yield unreliable rankings due to high variance.2324## Inputs2526- pqm_development: labeled peak quality metrics matrix with ground-truth pass/fail labels27- metric set specification (e.g., 'M11')28- random seed (integer, for reproducibility)29- cross-validation parameters (k, repNum)3031## Outputs3233- cross-validation fold results: trained models and hyperparameters for each algorithm and fold34- evaluation measures table: Pass_FScore, Fail_FScore, Accuracy, Pass_Precision, Pass_Recall, Fail_Precision, Fail_Recall per model and round35- bar plots: visual comparison of mean scores across algorithms36- CD (Friedman-Demšar) plots and rank matrices: statistical ranking of algorithms37- best-performing algorithm identifier with mean scores3839## How to apply4041Execute runCrossValidation on the labeled peak quality metrics matrix (pqm_development) with specified parameters: k=5 folds, repNum=10 repetitions, a fixed random seed (e.g., 453), your chosen metric set (e.g., 'M11'), and the full subset of algorithms to compare. Apply getEvaluationMeasures to extract Pass_FScore, Fail_FScore, Accuracy, and other measures across all cross-validation rounds. Visualize results using getBarPlots to compare mean performance scores across models, then generate CD (Friedman-Demšar) plots with getCDPlots (compareBest=FALSE) to obtain statistical rankings that account for multiple comparisons. Rank models by mean evaluation scores and statistical significance; the algorithm with the highest mean score and lowest critical distance is the best performer for your metric set.4243## Related tools4445- **MetaClean** (R package providing runCrossValidation, getEvaluationMeasures, getBarPlots, and getCDPlots functions for multi-algorithm comparison and statistical ranking) — https://github.com/KelseyChetnik/MetaClean46- **caret** (underlying machine learning framework in R used by MetaClean to implement all 8 classification algorithms)47- **MetaCleanData** (R package providing example development and test peak-quality metrics matrices for reproducible cross-validation studies) — https://github.com/KelseyChetnik/MetaCleanData48- **R** (runtime environment and language for executing MetaClean functions)4950## Examples5152```53runCrossValidation(pqm_development, k=5, repNum=10, rand.seed=453, algorithms=c('DecisionTree','LogisticRegression','NaiveBayes','RandomForest','SVM_Linear','AdaBoost','NeuralNetwork','ModelAveragedNeuralNetwork'), metricSet='M11')54```5556## Evaluation signals5758- Cross-validation completes without errors for all k·repNum fold splits and all requested algorithms; no missing hyperparameter sets returned.59- Evaluation measures are computed for all folds and all algorithms; no NaN or infinite values in Pass_FScore, Fail_FScore, or Accuracy columns.60- Bar plots show non-overlapping or clearly distinct mean scores across at least 2–3 top-ranked algorithms, indicating discriminative ranking.61- CD plot rankings are consistent with mean scores: algorithm with highest mean Pass_FScore appears at lowest critical distance rank.62- Friedman statistical test p-value (if reported) is <0.05, indicating significant differences among algorithms; if p≥0.05, no algorithm is truly superior within the metric set.6364## Limitations6566- Results are specific to the chosen metric set (e.g., M11); algorithm rankings may differ for other metric sets (e.g., M1–M8 or custom subsets).67- Cross-validation performance does not guarantee generalization to truly independent test data; final validation on held-out pqm_test is required before deployment.68- MetaClean uses only 8 fixed algorithms (DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM_Linear, AdaBoost, NeuralNetwork, ModelAveragedNeuralNetwork); custom or ensemble algorithms not in this set cannot be ranked using runCrossValidation.69- Hyperparameter tuning is performed by caret's internal defaults; systematic hyperparameter optimization is not described in the workflow and may affect algorithm rankings if different hyperparameters are chosen.7071## Evidence7273- [other] 5-fold cross-validation with 10 repetitions on M11 metric set using random seed 453: "running 5-fold cross-validation with 10 repetitions on the M11 metric set using MetaCleanData with random seed 453"74- [methods] runCrossValidation function wraps cross-validation training of user-selected algorithm subset: "The runCrossValidation function is a wrapper function that uses cross-validation to train a user-selected subset of the 8 available algorithms."75- [methods] getEvaluationMeasures extracts performance metrics per model across CV rounds: "We use the getEvaluationMeasures function to do this."76- [methods] getCDPlots generates statistical CD plots and rank matrices for algorithm comparison: "MetaClean also provides a wrapper function to easily generate CD plots of the evaluation measures for each model. This is getCDPlots"77- [other] AdaBoost identified as best algorithm by comparing mean scores and statistical rankings: "Identify AdaBoost as the best-performing algorithm by comparing mean evaluation scores and statistical rankings across all models."78- [readme] 9 diverse machine learning algorithms available for training and comparison: "MetaClean utilizes 12 peak-quality metrics and 9 diverse machine learning algorithms to build a classifier to detect and flag low-quality peaks in untargeted metabolomics data."
HolobiomicsLab 