prediction-error-analysis
License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution.
Summary
Comparative evaluation of machine learning model predictions by computing prediction errors (mean absolute error, median absolute error) across different feature sets or model configurations on a held-out test set. This skill enables quantitative assessment of whether different molecular representations (descriptors, fingerprints, or combinations) produce meaningfully different retention time prediction accuracy.
When to use
When you have trained multiple machine learning regressors on the same prediction task (e.g., retention time prediction) using different feature configurations or molecular representations, and you need to determine which configuration produces the lowest prediction errors and whether the differences are meaningful. Specifically applicable when comparing descriptor-only, fingerprint-only, or combined feature sets for metabolite annotation or retention time forecasting.
When NOT to use
- Training data and test data overlap or are not independently sampled — error estimates will be unreliably optimistic
- Only one model or feature set has been trained; no meaningful comparison is possible
- Input is already a feature table or preprocessed descriptor/fingerprint matrix rather than raw molecular structures — error analysis requires end-to-end trained models
Inputs
- trained machine learning regressors (one per feature set or configuration)
- held-out test set with molecular structures and experimental retention times
- model predictions on test set (continuous values)
Outputs
- mean absolute error (MAE) per model configuration with ± standard deviation
- median absolute error (MedAE) per model configuration with ± standard deviation
- ranked comparison table of feature sets or model architectures
- error distribution visualization (e.g., histogram or box plot across configurations)
How to apply
Train independent machine learning regressors on each feature set or model configuration using the same training data and hyperparameters. Apply each trained model to a held-out test set containing ground-truth labels (e.g., experimental retention times). For each model, compute mean absolute error (MAE) and median absolute error (MedAE) by comparing predictions to observed values. Compare error distributions across all configurations and rank them by both central tendency and dispersion (±standard deviation). Use these metrics to identify the feature set or model that achieves lowest error and most stable predictions. The rationale is that fingerprints often encode structural patterns more effectively than raw descriptors for chromatographic properties, but empirical error comparison on your specific dataset and test split is the only valid criterion.
Related tools
- alvaDesc (Generate molecular descriptors and fingerprints (MACCS166, Extended Connectivity, Path Fingerprints) that serve as input feature sets for training regressors prior to error analysis) — https://www.alvascience.com/alvadesc/
- RDKit (Alternative fingerprint generation tool for training machine learning models whose predictions are then compared via error metrics)
Examples
python cmmrt/rt/train_model.py --storage sqlite:///results/optuna/train.db --save_to saved_models; python cmmrt/rt/test_model.py --model_dir saved_models --test_data SMRT_test.csv --output_errors rt_errors.csv
Evaluation signals
- Error metrics (MAE, MedAE) computed on a held-out test set independent of training and hyperparameter tuning splits
- Standard deviation reported alongside mean and median errors; variability should be small relative to central value (e.g., ±1–3 s for retention time in seconds)
- Ranking of configurations is consistent across both MAE and MedAE; if fingerprints outperform descriptors, both metrics should show the same trend
- Error distributions should show fingerprint-only and combined models producing lower error quantiles (25th, 50th, 75th percentiles) than descriptor-only models
- Comparison is performed on the same test set and same models; no retraining or data leakage between configurations
Limitations
- Error comparison is sensitive to train/test split; nested cross-validation is recommended for robust estimates, but single held-out test results may be unstable with small test sets
- The superiority of fingerprints over descriptors observed in this study (METLIN SMRT, 80,038 molecules, MACCS/ECFP/PFP) may not generalize to other datasets, chromatographic methods, or molecular domains with different structural diversity
- Requires that all competing models are trained with the same hyperparameters and stopping criteria; unfair hyperparameter tuning per configuration will bias error comparisons
- Mean and median absolute error do not capture prediction uncertainty; complementary methods (e.g., quantile regression, Bayesian predictive intervals) are needed to assess whether prediction confidence varies across feature sets
Evidence
- [intro] fingerprints tend to perform better than descriptors alone or combined: "Results suggest that fingerprints tend to perform better."
- [intro] MAE and MedAE as error metrics: "Compute prediction errors (mean absolute error and median absolute error) for each of the three trained models on a held-out test set."
- [intro] three feature configurations compared independently: "Train machine learning regressors independently on three feature sets: (a) descriptors only, (b) fingerprints only, (c) both descriptors and fingerprints combined."
- [intro] benchmark results on METLIN SMRT: "The best results were obtained by a heavily regularized DNN trained with cosine annealing warm restarts and stochastic weight averaging, achieving a mean and median absolute errors of 39.2±1.2 s and"
- [readme] fingerprint types used for comparison: "5,666 molecular descriptors and 2,214 fingerprints (MACCS166, Extended Connectivity, and Path Fingerprints fingerprints) were generated with the alvaDesc software. The models were trained using only"
1---2name: prediction-error-analysis3description: Use when when you have trained multiple machine learning regressors on the same prediction task (e.4license: CC-BY-4.05---67# prediction-error-analysis89> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->10## Summary1112Comparative evaluation of machine learning model predictions by computing prediction errors (mean absolute error, median absolute error) across different feature sets or model configurations on a held-out test set. This skill enables quantitative assessment of whether different molecular representations (descriptors, fingerprints, or combinations) produce meaningfully different retention time prediction accuracy.1314## When to use1516When you have trained multiple machine learning regressors on the same prediction task (e.g., retention time prediction) using different feature configurations or molecular representations, and you need to determine which configuration produces the lowest prediction errors and whether the differences are meaningful. Specifically applicable when comparing descriptor-only, fingerprint-only, or combined feature sets for metabolite annotation or retention time forecasting.1718## When NOT to use1920- Training data and test data overlap or are not independently sampled — error estimates will be unreliably optimistic21- Only one model or feature set has been trained; no meaningful comparison is possible22- Input is already a feature table or preprocessed descriptor/fingerprint matrix rather than raw molecular structures — error analysis requires end-to-end trained models2324## Inputs2526- trained machine learning regressors (one per feature set or configuration)27- held-out test set with molecular structures and experimental retention times28- model predictions on test set (continuous values)2930## Outputs3132- mean absolute error (MAE) per model configuration with ± standard deviation33- median absolute error (MedAE) per model configuration with ± standard deviation34- ranked comparison table of feature sets or model architectures35- error distribution visualization (e.g., histogram or box plot across configurations)3637## How to apply3839Train independent machine learning regressors on each feature set or model configuration using the same training data and hyperparameters. Apply each trained model to a held-out test set containing ground-truth labels (e.g., experimental retention times). For each model, compute mean absolute error (MAE) and median absolute error (MedAE) by comparing predictions to observed values. Compare error distributions across all configurations and rank them by both central tendency and dispersion (±standard deviation). Use these metrics to identify the feature set or model that achieves lowest error and most stable predictions. The rationale is that fingerprints often encode structural patterns more effectively than raw descriptors for chromatographic properties, but empirical error comparison on your specific dataset and test split is the only valid criterion.4041## Related tools4243- **alvaDesc** (Generate molecular descriptors and fingerprints (MACCS166, Extended Connectivity, Path Fingerprints) that serve as input feature sets for training regressors prior to error analysis) — https://www.alvascience.com/alvadesc/44- **RDKit** (Alternative fingerprint generation tool for training machine learning models whose predictions are then compared via error metrics)4546## Examples4748```49python cmmrt/rt/train_model.py --storage sqlite:///results/optuna/train.db --save_to saved_models; python cmmrt/rt/test_model.py --model_dir saved_models --test_data SMRT_test.csv --output_errors rt_errors.csv50```5152## Evaluation signals5354- Error metrics (MAE, MedAE) computed on a held-out test set independent of training and hyperparameter tuning splits55- Standard deviation reported alongside mean and median errors; variability should be small relative to central value (e.g., ±1–3 s for retention time in seconds)56- Ranking of configurations is consistent across both MAE and MedAE; if fingerprints outperform descriptors, both metrics should show the same trend57- Error distributions should show fingerprint-only and combined models producing lower error quantiles (25th, 50th, 75th percentiles) than descriptor-only models58- Comparison is performed on the same test set and same models; no retraining or data leakage between configurations5960## Limitations6162- Error comparison is sensitive to train/test split; nested cross-validation is recommended for robust estimates, but single held-out test results may be unstable with small test sets63- The superiority of fingerprints over descriptors observed in this study (METLIN SMRT, 80,038 molecules, MACCS/ECFP/PFP) may not generalize to other datasets, chromatographic methods, or molecular domains with different structural diversity64- Requires that all competing models are trained with the same hyperparameters and stopping criteria; unfair hyperparameter tuning per configuration will bias error comparisons65- Mean and median absolute error do not capture prediction uncertainty; complementary methods (e.g., quantile regression, Bayesian predictive intervals) are needed to assess whether prediction confidence varies across feature sets6667## Evidence6869- [intro] fingerprints tend to perform better than descriptors alone or combined: "Results suggest that fingerprints tend to perform better."70- [intro] MAE and MedAE as error metrics: "Compute prediction errors (mean absolute error and median absolute error) for each of the three trained models on a held-out test set."71- [intro] three feature configurations compared independently: "Train machine learning regressors independently on three feature sets: (a) descriptors only, (b) fingerprints only, (c) both descriptors and fingerprints combined."72- [intro] benchmark results on METLIN SMRT: "The best results were obtained by a heavily regularized DNN trained with cosine annealing warm restarts and stochastic weight averaging, achieving a mean and median absolute errors of 39.2±1.2 s and"73- [readme] fingerprint types used for comparison: "5,666 molecular descriptors and 2,214 fingerprints (MACCS166, Extended Connectivity, and Path Fingerprints fingerprints) were generated with the alvaDesc software. The models were trained using only"
HolobiomicsLab