Model Performance Evaluation Roc Curves

model-performance-evaluation-roc-curves

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution.

Summary

Evaluate and compare binary classification model performance by training logistic regression, random forest, and XGBoost models on metabolomics data under cross-validation and recording performance metrics and ROC curves for each algorithm. This skill is essential when assessing which supervised learning approach best discriminates between two metabolic phenotypes.

When to use

Apply this skill when you have a binary classification task on metabolomics data (e.g., covid_data) and need to benchmark multiple algorithms to determine which produces the highest discriminative power. Use it after data preprocessing, imputation, and normalization are complete, and when you have defined a categorical outcome variable (e.g., disease status, treatment response) and selected your metabolite feature set.

When NOT to use

• Input outcome variable is not binary (use multi-class classification methods instead)
• Metabolomics data has not been imputed and normalized (preprocess first)
• Test set size is 0% and ROC curves are not required (use training metrics only if appropriate for your research question)

Inputs

• metabolomics dataset (e.g., covid_data) in MeTEor-compatible format
• categorical grouping variable (binary outcome)
• selected metabolite features (numeric columns)
• cross-validation configuration (fold count, stratification)

Outputs

• logistic regression performance metrics and ROC curve
• random forest performance metrics and ROC curve
• XGBoost performance metrics and ROC curve
• top ten most important features per model
• comparative performance summary across algorithms

How to apply

Within MeTEor's Prediction module, configure the cross-validation scheme (stratified 5-fold or leave-one-out) and test set size (>0% to generate ROC curves). Train each of the three algorithms sequentially—logistic regression, random forest, and XGBoost—recording performance metrics (accuracy, AUC, sensitivity, specificity) and ROC curves for each. Retrieve and document the ten most important features identified by each model to assess feature consistency and biological plausibility. Compare performance metrics across algorithms to identify the best-performing model; models trained on the same cross-validation scheme with the same test/train split ensure fair comparison.

Related tools

• MeTEor (R Shiny application implementing logistic regression, random forest, and XGBoost prediction models with cross-validation and ROC curve visualization for binary classification on metabolomics data) — https://github.com/scibiome/meteor
• R (Statistical computing environment in which MeTEor runs)

Examples

library(MeTEor); meteor() # Launch MeTEor app → load covid_data via Configurator → navigate to Prediction module → set cross-validation to 'stratified 5-fold', test set size to 20% → train Logistic Regression, Random Forest, XGBoost sequentially → record ROC curves and feature importance

Evaluation signals

• ROC curves are generated for each algorithm (when test set size > 0%); curves should show typical sigmoid shape with AUC ≥ 0.5
• Performance metrics (accuracy, AUC, sensitivity, specificity) are recorded and comparable across all three algorithms using the same cross-validation scheme
• Top ten features are retrieved and documented for each model; overlapping features across models suggest robust biomarkers
• Cross-validation folds and test/train splits are consistent across all three model trainings to ensure fair comparison
• No algorithm performance metrics are missing; if a metric cannot be computed, document the reason (e.g., single class in a fold)

Limitations

• ROC curves require test set size > 0%; when test set size = 0%, ROC curves are not generated and only cross-validation metrics are available
• Class imbalance in the binary outcome may inflate or deflate performance metrics; stratified cross-validation mitigates this but does not eliminate the issue
• Feature importance rankings differ across algorithms (logistic regression uses coefficient magnitude, random forest and XGBoost use tree-based importance); direct comparison of importance scores across models is not recommended
• MeTEor is designed for longitudinal metabolomics data; non-temporal or cross-sectional metabolomics may not fully utilize all application features

Evidence

• [other] Train logistic regression (LR) model and record performance metrics and ROC curve (if test set size > 0%): "Train logistic regression (LR) model and record performance metrics and ROC curve (if test set size > 0%)."
• [other] MeTEor implements three classification algorithms—logistic regression (LR), random forest (RF), and extreme gradient boosting (XGB): "MeTEor implements three classification algorithms—logistic regression (LR), random forest (RF), and extreme gradient boosting (XGB)—for binary classification tasks on metabolomics data such as"
• [other] Retrieve and record the ten most important features identified by the models (or fewer if not all features are used): "Retrieve and record the ten most important features identified by the models (or fewer if not all features are used)."
• [other] Select test set size and cross-validation type (stratified 5-fold cross-validation or leave-one-out cross-validation): "Access the Prediction module and select test set size and cross-validation type (stratified 5-fold cross-validation or leave-one-out cross-validation)."
• [readme] Prediction models: Logistic regression, Random Forest, XGBoost: "Prediction models: Logistic regression, Random Forest, XGBoost"