Statistical Model Ranking And Selection

statistical-model-ranking-and-selection

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution.

Summary

Rank and select the best-performing machine learning classifier from a candidate set using cross-validation, evaluation metrics (F-score, precision, recall, accuracy), and statistical comparison plots (Friedman-Demšar CD plots). This skill identifies the optimal algorithm for a specific downstream task (e.g., peak quality classification) by comparing mean performance scores and statistical significance across all folds and repetitions.

When to use

You have trained multiple machine learning algorithms (e.g., DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM, AdaBoost, NeuralNetwork) on the same labeled dataset using k-fold cross-validation with multiple repetitions, and you need to select a single best model for deployment or further tuning. Typical trigger: you have cross-validation results for ≥3 algorithms and need to decide which to train as the final classifier.

When NOT to use

• Only one algorithm has been trained; statistical ranking requires ≥2 candidate models.
• Cross-validation has not been completed; this skill operates on already-trained CV results, not raw data.
• You have only a single train-test split (not repeated cross-validation); statistical comparison via CD plots requires variance estimates across multiple folds.

Inputs

• cross-validation results: list of trained models, hyperparameters, and predictions from runCrossValidation()
• peak quality metrics matrix (pqm_development) with k=5 folds and repNum=10 repetitions
• metric set specification (e.g., 'M11')
• algorithm specification (list of ≥2 candidate algorithms)

Outputs

• evaluation measures data frame: Pass_FScore, Fail_FScore, Accuracy, Pass_Precision, Pass_Recall, Fail_Precision, Fail_Recall per algorithm and fold
• bar plots visualizing mean evaluation measures (Pass_FScore, Fail_FScore, Accuracy) by algorithm
• Friedman-Demšar CD plots showing statistical rankings and significance groups
• rank matrix identifying best-performing algorithm and confidence in selection
• selected algorithm name (e.g., 'AdaBoost') to pass to trainClassifier() for final model training

How to apply

After running cross-validation across all candidate algorithms, calculate evaluation measures (Pass_FScore, Fail_FScore, Accuracy, precision, recall) for each model across all folds and repetitions using getEvaluationMeasures(). Visualize performance using getBarPlots() to identify candidate leaders. Then generate statistical ranking plots using getCDPlots() with the Friedman-Demšar test (with compareBest=FALSE) to rank models within the metric set and assess whether performance differences are statistically significant. Compare mean evaluation scores and statistical rankings to identify the best-performing algorithm, which becomes the basis for training the final classifier. This approach accounts for variance across cross-validation rounds rather than relying on a single train-test split.

Related tools

• MetaClean (Provides runCrossValidation(), getEvaluationMeasures(), getBarPlots(), getCDPlots() wrapper functions to execute cross-validation, compute evaluation metrics, and generate statistical ranking visualizations.) — https://github.com/KelseyChetnik/MetaClean
• caret (Underlying R package implementing the 8 machine learning algorithms (DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM_Linear, AdaBoost, NeuralNetwork, ModelAveragedNeuralNetwork) trained and compared within MetaClean's cross-validation framework.)
• R (Statistical computing environment in which MetaClean, caret, and CD plot generation (Friedman-Demšar test) are executed.)

Examples

# Compute evaluation measures for all cross-validated models
eval_measures <- getEvaluationMeasures(cv_results)

# Visualize Pass_FScore, Fail_FScore, Accuracy by algorithm
getBarPlots(eval_measures, measures = c('Pass_FScore', 'Fail_FScore', 'Accuracy'))

# Generate Friedman-Demšar CD plots and rank models
getCDPlots(eval_measures, compareBest = FALSE)

# Extract best algorithm from rank matrix
best_algo <- names(which.min(colMeans(rank_matrix)))

Evaluation signals

• Evaluation measures (Pass_FScore, Fail_FScore, Accuracy) are computed for every model across all k folds and all repetitions; no fold or algorithm is missing.
• Bar plots show non-zero, bounded values (F-scores and accuracy in [0,1]); extreme or NaN values indicate failed cross-validation.
• CD plots render statistical groupings (critical difference intervals) such that algorithms within the same group are not significantly different (α=0.05 typically); algorithms outside are ranked as significantly better or worse.
• Rank matrix displays all algorithms with average rank scores; best algorithm has the lowest mean rank and is positioned at the left edge of the CD plot.
• Selected algorithm name matches one of the candidate algorithms passed to runCrossValidation(); verify the selection is consistent with both bar plot and CD plot rankings.

Limitations

• Statistical power depends on the number of folds and repetitions; 5 folds with 10 repetitions (50 total evaluations per algorithm) provides moderate power but may miss subtle differences in small datasets.
• CD plots assume exchangeable cross-validation rounds and do not account for temporal or data dependency structure; if folds are not independent, ranking may be biased.
• Metric set selection ('M11', etc.) is fixed before cross-validation; comparison is valid only within one metric set, not across different feature sets.
• Final model selection based on training-set cross-validation performance may not generalize to an independent test set; post-selection validation on a held-out test set is recommended but not included in this skill.

Evidence

• [methods] Apply getEvaluationMeasures to calculate Pass_FScore, Pass_Precision, Pass_Recall, Fail_FScore, Fail_Precision, Fail_Recall, and Accuracy across all cross-validation rounds and models.: "Apply getEvaluationMeasures to calculate Pass_FScore, Pass_Precision, Pass_Recall, Fail_FScore, Fail_Precision, Fail_Recall, and Accuracy across all cross-validation rounds and models."
• [methods] Visualize evaluation measures using getBarPlots (Pass_FScore, Fail_FScore, Accuracy) to assess model performance.: "Visualize evaluation measures using getBarPlots (Pass_FScore, Fail_FScore, Accuracy) to assess model performance."
• [methods] Generate CD (Friedman-Demšar) plots and rank matrices using getCDPlots with compareBest=FALSE to statistically rank models within the M11 metric set.: "Generate CD (Friedman-Demšar) plots and rank matrices using getCDPlots with compareBest=FALSE to statistically rank models within the M11 metric set."
• [methods] Identify AdaBoost as the best-performing algorithm by comparing mean evaluation scores and statistical rankings across all models.: "Identify AdaBoost as the best-performing algorithm by comparing mean evaluation scores and statistical rankings across all models."
• [readme] MetaClean utilizes 12 peak-quality metrics and 9 diverse machine learning algorithms to build a classifier to detect and flag low-quality peaks in untargeted metabolomics data: "MetaClean utilizes 12 peak-quality metrics and 9 diverse machine learning algorithms to build a classifier to detect and flag low-quality peaks in untargeted metabolomics data"