# Statistical Model Ranking And Selection

> Use when you have trained multiple machine learning algorithms (e.

- Skill: `holobiomicslab/statistical-model-ranking-and-selection` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/statistical-model-ranking-and-selection`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/statistical-model-ranking-and-selection/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/statistical-model-ranking-and-selection

---


# statistical-model-ranking-and-selection

> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->
## Summary

Rank and select the best-performing machine learning classifier from a candidate set using cross-validation, evaluation metrics (F-score, precision, recall, accuracy), and statistical comparison plots (Friedman-Demšar CD plots). This skill identifies the optimal algorithm for a specific downstream task (e.g., peak quality classification) by comparing mean performance scores and statistical significance across all folds and repetitions.

## When to use

You have trained multiple machine learning algorithms (e.g., DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM, AdaBoost, NeuralNetwork) on the same labeled dataset using k-fold cross-validation with multiple repetitions, and you need to select a single best model for deployment or further tuning. Typical trigger: you have cross-validation results for ≥3 algorithms and need to decide which to train as the final classifier.

## When NOT to use

- Only one algorithm has been trained; statistical ranking requires ≥2 candidate models.
- Cross-validation has not been completed; this skill operates on already-trained CV results, not raw data.
- You have only a single train-test split (not repeated cross-validation); statistical comparison via CD plots requires variance estimates across multiple folds.

## Inputs

- cross-validation results: list of trained models, hyperparameters, and predictions from runCrossValidation()
- peak quality metrics matrix (pqm_development) with k=5 folds and repNum=10 repetitions
- metric set specification (e.g., 'M11')
- algorithm specification (list of ≥2 candidate algorithms)

## Outputs

- evaluation measures data frame: Pass_FScore, Fail_FScore, Accuracy, Pass_Precision, Pass_Recall, Fail_Precision, Fail_Recall per algorithm and fold
- bar plots visualizing mean evaluation measures (Pass_FScore, Fail_FScore, Accuracy) by algorithm
- Friedman-Demšar CD plots showing statistical rankings and significance groups
- rank matrix identifying best-performing algorithm and confidence in selection
- selected algorithm name (e.g., 'AdaBoost') to pass to trainClassifier() for final model training

## How to apply

After running cross-validation across all candidate algorithms, calculate evaluation measures (Pass_FScore, Fail_FScore, Accuracy, precision, recall) for each model across all folds and repetitions using getEvaluationMeasures(). Visualize performance using getBarPlots() to identify candidate leaders. Then generate statistical ranking plots using getCDPlots() with the Friedman-Demšar test (with compareBest=FALSE) to rank models within the metric set and assess whether performance differences are statistically significant. Compare mean evaluation scores and statistical rankings to identify the best-performing algorithm, which becomes the basis for training the final classifier. This approach accounts for variance across cross-validation rounds rather than relying on a single train-test split.

## Related tools

- **MetaClean** (Provides runCrossValidation(), getEvaluationMeasures(), getBarPlots(), getCDPlots() wrapper functions to execute cross-validation, compute evaluation metrics, and generate statistical ranking visualizations.) — https://github.com/KelseyChetnik/MetaClean
- **caret** (Underlying R package implementing the 8 machine learning algorithms (DecisionTree, LogisticRegression, NaiveBayes, RandomForest, SVM_Linear, AdaBoost, NeuralNetwork, ModelAveragedNeuralNetwork) trained and compared within MetaClean's cross-validation framework.)
- **R** (Statistical computing environment in which MetaClean, caret, and CD plot generation (Friedman-Demšar test) are executed.)

## Examples

```
# Compute evaluation measures for all cross-validated models
eval_measures <- getEvaluationMeasures(cv_results)

# Visualize Pass_FScore, Fail_FScore, Accuracy by algorithm
getBarPlots(eval_measures, measures = c('Pass_FScore', 'Fail_FScore', 'Accuracy'))

# Generate Friedman-Demšar CD plots and rank models
getCDPlots(eval_measures, compareBest = FALSE)

# Extract best algorithm from rank matrix
best_algo <- names(which.min(colMeans(rank_matrix)))
```

## Evaluation signals

- Evaluation measures (Pass_FScore, Fail_FScore, Accuracy) are computed for every model across all k folds and all repetitions; no fold or algorithm is missing.
- Bar plots show non-zero, bounded values (F-scores and accuracy in [0,1]); extreme or NaN values indicate failed cross-validation.
- CD plots render statistical groupings (critical difference intervals) such that algorithms within the same group are not significantly different (α=0.05 typically); algorithms outside are ranked as significantly better or worse.
- Rank matrix displays all algorithms with average rank scores; best algorithm has the lowest mean rank and is positioned at the left edge of the CD plot.
- Selected algorithm name matches one of the candidate algorithms passed to runCrossValidation(); verify the selection is consistent with both bar plot and CD plot rankings.

## Limitations

- Statistical power depends on the number of folds and repetitions; 5 folds with 10 repetitions (50 total evaluations per algorithm) provides moderate power but may miss subtle differences in small datasets.
- CD plots assume exchangeable cross-validation rounds and do not account for temporal or data dependency structure; if folds are not independent, ranking may be biased.
- Metric set selection ('M11', etc.) is fixed before cross-validation; comparison is valid only within one metric set, not across different feature sets.
- Final model selection based on training-set cross-validation performance may not generalize to an independent test set; post-selection validation on a held-out test set is recommended but not included in this skill.

## Evidence

- [methods] Apply getEvaluationMeasures to calculate Pass_FScore, Pass_Precision, Pass_Recall, Fail_FScore, Fail_Precision, Fail_Recall, and Accuracy across all cross-validation rounds and models.: "Apply getEvaluationMeasures to calculate Pass_FScore, Pass_Precision, Pass_Recall, Fail_FScore, Fail_Precision, Fail_Recall, and Accuracy across all cross-validation rounds and models."
- [methods] Visualize evaluation measures using getBarPlots (Pass_FScore, Fail_FScore, Accuracy) to assess model performance.: "Visualize evaluation measures using getBarPlots (Pass_FScore, Fail_FScore, Accuracy) to assess model performance."
- [methods] Generate CD (Friedman-Demšar) plots and rank matrices using getCDPlots with compareBest=FALSE to statistically rank models within the M11 metric set.: "Generate CD (Friedman-Demšar) plots and rank matrices using getCDPlots with compareBest=FALSE to statistically rank models within the M11 metric set."
- [methods] Identify AdaBoost as the best-performing algorithm by comparing mean evaluation scores and statistical rankings across all models.: "Identify AdaBoost as the best-performing algorithm by comparing mean evaluation scores and statistical rankings across all models."
- [readme] MetaClean utilizes 12 peak-quality metrics and 9 diverse machine learning algorithms to build a classifier to detect and flag low-quality peaks in untargeted metabolomics data: "MetaClean utilizes 12 peak-quality metrics and 9 diverse machine learning algorithms to build a classifier to detect and flag low-quality peaks in untargeted metabolomics data"