metabolite-retention-time-comparison
Summary
Compare predicted liquid-chromatography retention orders against experimentally determined ground-truth orderings using ranking accuracy and correlation metrics. This skill validates retention-order prediction models on metabolite datasets by quantifying how well predicted orderings match observed LC retention sequences.
When to use
After executing retention-order prediction on a test set of metabolites, you have both predicted retention orderings and experimentally measured ground-truth retention orderings for the same set of compounds, and you need to quantify prediction accuracy to validate the model's performance across different LC systems or training scenarios.
When NOT to use
- Ground-truth retention orderings are not available or are incomplete (missing pairwise comparisons for evaluation)
- Predicted orderings have not yet been generated by a model (comparison requires both predictions and observations)
- Input metabolite dataset is from a single LC system and no cross-system evaluation is intended (use only for single-system validation, not transfer-learning assessment)
Inputs
- predicted retention orders (JSON or structured array per target system and scenario)
- ground-truth experimental retention orderings (metabolite compound pairs with known retention rank)
- configuration file specifying experiment scenario, molecular features, and kernel choice
Outputs
- results summary table with predicted vs. ground-truth retention orders and performance scores
- ranking accuracy and Spearman correlation coefficient per target system
- structured results directory organized by scenario, feature representation, and kernel type
How to apply
Load the predicted retention orders and ground-truth orderings from the experiment output (typically stored in structured result directories by scenario). Apply ranking-based evaluation metrics—specifically ranking accuracy (fraction of correctly predicted pairwise orderings) and Spearman rank correlation (monotonic association between predicted and observed ranks)—to each target system individually. Generate a results summary table listing predicted vs. ground-truth retention orders and corresponding performance scores for each system and scenario. Use these metrics to assess generalization: single-system predictions (baseline_single scenario) serve as a lower-bound reference, while cross-system predictions (all_on_one scenario, with and without leave-target-system-out flag) demonstrate transfer capability.
Related tools
- evaluation_scenarios_main.py (Orchestrates experiment execution and produces predicted retention orders for specified scenario, estimator, and target system; outputs are stored hierarchically by feature representation and kernel, then compared against ground truth) — github.com/aalto-ics-kepaco/retention_order_prediction
- RankSVM (Order predictor estimator used to generate pairwise retention orderings; predictions are fed into comparison workflow) — github.com/aalto-ics-kepaco/retention_order_prediction
- SVR (Support Vector Regression) (Alternative order predictor estimator option; generates retention scores that can be converted to orderings for comparison) — github.com/aalto-ics-kepaco/retention_order_prediction
- scipy, numpy, pandas (Python packages for numerical computation, ranking calculations, and tabular results organization)
- scikit-learn (Provides ranking metrics and correlation functions (Spearman) used in evaluation)
Examples
python src/evaluation_scenarios_main.py ranksvm baseline_single 10 -1 results/raw/PredRet/v2/config.json 2 False
Evaluation signals
- Ranking accuracy is non-negative and ≤ 1.0 (fraction of correctly ordered pairs); baseline_single scenarios typically show 0.50–0.95 range depending on system and feature representation
- Spearman rank correlation coefficient ranges from −1 to +1; for well-performing models on same-system prediction, expect values > 0.7; cross-system transfer typically yields lower correlations (0.3–0.7)
- Results summary table is complete (no missing entries) for all target systems in the specified scenario; rows correspond to unique metabolite pairs, columns to predicted rank, ground-truth rank, and performance metrics
- Output directory structure matches expected hierarchy:
results/PredRet/v2/final/{estimator}_{params}/{feature_representation}/{kernel}/{scenario}
- Performance comparison shows expected ranking: baseline_single ≥ all_on_one (LTSO=False) ≥ all_on_one (LTSO=True), reflecting increasing domain shift; deviation suggests data leakage or misconfigured split
Limitations
- Evaluation is restricted to pairwise relative orderings; absolute retention-time prediction (e.g., minutes) is not assessed
- Performance depends heavily on choice of molecular features (MACCS fingerprints binary vs. counting) and kernel function (minmax, tanimoto); comparison results are not portable across feature/kernel combinations without rerunning the full pipeline
- Cross-system transfer (all_on_one scenario) assumes that training systems are sufficiently similar; highly dissimilar LC systems or unusual mobile phases may show poor generalization despite high baseline_single accuracy
- Ranking metrics assume a fixed set of metabolites per LC system; results are not comparable across datasets with different metabolite sets or different numbers of ground-truth orderings
Evidence
- [other] Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar).: "Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar)."
- [readme] Single system used as training and target. Different percentage of data used for trainging.: "Single system used as training and target. Different percentage of data used for trainging."
- [readme] All systems used for training. Single system used as target. Target system in training (LTSO): True & False: "All systems used for training. Single system used as target. Target system in training (LTSO): True & False"
- [readme] The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └── maccsCount_f2dcf0b3 └── minmax └── baseline_single: "The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └──"
- [other] Generate a results summary table or report listing predicted vs. ground-truth retention orders and performance scores.: "Generate a results summary table or report listing predicted vs. ground-truth retention orders and performance scores."
1---2name: metabolite-retention-time-comparison3description: Use when after executing retention-order prediction on a test set of metabolites, you have both predicted retention orderings and experimentally measured ground-truth retention orderings for the same set of compounds, and you need to quantify prediction accuracy to validate the model's performance.4license: CC-BY-4.05---67# metabolite-retention-time-comparison89## Summary1011Compare predicted liquid-chromatography retention orders against experimentally determined ground-truth orderings using ranking accuracy and correlation metrics. This skill validates retention-order prediction models on metabolite datasets by quantifying how well predicted orderings match observed LC retention sequences.1213## When to use1415After executing retention-order prediction on a test set of metabolites, you have both predicted retention orderings and experimentally measured ground-truth retention orderings for the same set of compounds, and you need to quantify prediction accuracy to validate the model's performance across different LC systems or training scenarios.1617## When NOT to use1819- Ground-truth retention orderings are not available or are incomplete (missing pairwise comparisons for evaluation)20- Predicted orderings have not yet been generated by a model (comparison requires both predictions and observations)21- Input metabolite dataset is from a single LC system and no cross-system evaluation is intended (use only for single-system validation, not transfer-learning assessment)2223## Inputs2425- predicted retention orders (JSON or structured array per target system and scenario)26- ground-truth experimental retention orderings (metabolite compound pairs with known retention rank)27- configuration file specifying experiment scenario, molecular features, and kernel choice2829## Outputs3031- results summary table with predicted vs. ground-truth retention orders and performance scores32- ranking accuracy and Spearman correlation coefficient per target system33- structured results directory organized by scenario, feature representation, and kernel type3435## How to apply3637Load the predicted retention orders and ground-truth orderings from the experiment output (typically stored in structured result directories by scenario). Apply ranking-based evaluation metrics—specifically ranking accuracy (fraction of correctly predicted pairwise orderings) and Spearman rank correlation (monotonic association between predicted and observed ranks)—to each target system individually. Generate a results summary table listing predicted vs. ground-truth retention orders and corresponding performance scores for each system and scenario. Use these metrics to assess generalization: single-system predictions (baseline_single scenario) serve as a lower-bound reference, while cross-system predictions (all_on_one scenario, with and without leave-target-system-out flag) demonstrate transfer capability.3839## Related tools4041- **evaluation_scenarios_main.py** (Orchestrates experiment execution and produces predicted retention orders for specified scenario, estimator, and target system; outputs are stored hierarchically by feature representation and kernel, then compared against ground truth) — github.com/aalto-ics-kepaco/retention_order_prediction42- **RankSVM** (Order predictor estimator used to generate pairwise retention orderings; predictions are fed into comparison workflow) — github.com/aalto-ics-kepaco/retention_order_prediction43- **SVR (Support Vector Regression)** (Alternative order predictor estimator option; generates retention scores that can be converted to orderings for comparison) — github.com/aalto-ics-kepaco/retention_order_prediction44- **scipy, numpy, pandas** (Python packages for numerical computation, ranking calculations, and tabular results organization)45- **scikit-learn** (Provides ranking metrics and correlation functions (Spearman) used in evaluation)4647## Examples4849```50python src/evaluation_scenarios_main.py ranksvm baseline_single 10 -1 results/raw/PredRet/v2/config.json 2 False51```5253## Evaluation signals5455- Ranking accuracy is non-negative and ≤ 1.0 (fraction of correctly ordered pairs); baseline_single scenarios typically show 0.50–0.95 range depending on system and feature representation56- Spearman rank correlation coefficient ranges from −1 to +1; for well-performing models on same-system prediction, expect values > 0.7; cross-system transfer typically yields lower correlations (0.3–0.7)57- Results summary table is complete (no missing entries) for all target systems in the specified scenario; rows correspond to unique metabolite pairs, columns to predicted rank, ground-truth rank, and performance metrics58- Output directory structure matches expected hierarchy: `results/PredRet/v2/final/{estimator}_{params}/{feature_representation}/{kernel}/{scenario}`59- Performance comparison shows expected ranking: baseline_single ≥ all_on_one (LTSO=False) ≥ all_on_one (LTSO=True), reflecting increasing domain shift; deviation suggests data leakage or misconfigured split6061## Limitations6263- Evaluation is restricted to pairwise relative orderings; absolute retention-time prediction (e.g., minutes) is not assessed64- Performance depends heavily on choice of molecular features (MACCS fingerprints binary vs. counting) and kernel function (minmax, tanimoto); comparison results are not portable across feature/kernel combinations without rerunning the full pipeline65- Cross-system transfer (all_on_one scenario) assumes that training systems are sufficiently similar; highly dissimilar LC systems or unusual mobile phases may show poor generalization despite high baseline_single accuracy66- Ranking metrics assume a fixed set of metabolites per LC system; results are not comparable across datasets with different metabolite sets or different numbers of ground-truth orderings6768## Evidence6970- [other] Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar).: "Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar)."71- [readme] Single system used as training and target. Different percentage of data used for trainging.: "Single system used as training and target. Different percentage of data used for trainging."72- [readme] All systems used for training. Single system used as target. Target system in training (LTSO): True & False: "All systems used for training. Single system used as target. Target system in training (LTSO): True & False"73- [readme] The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └── maccsCount_f2dcf0b3 └── minmax └── baseline_single: "The results will be stored into: results/PredRet/v2 └── final └── ranksvm_slacktype=on_pairs └── allow_overlap=True_d_lower=0_d_upper=16_ireverse=False_type=order_graph └── difference └──"74- [other] Generate a results summary table or report listing predicted vs. ground-truth retention orders and performance scores.: "Generate a results summary table or report listing predicted vs. ground-truth retention orders and performance scores."
HolobiomicsLab