ranking-performance-evaluation
Summary
Evaluate retention-order prediction models by comparing predicted rankings against ground-truth orderings using ranking-specific metrics such as ranking accuracy and Spearman correlation. This skill validates whether a learned order predictor (e.g., RankSVM) correctly recovers the relative retention order of metabolites in liquid chromatography.
When to use
Apply this skill after running retention-order prediction experiments on a test or held-out evaluation dataset. Use it when you have both predicted retention orders (from a trained model) and ground-truth retention orders (from experimental or reference data), and you need to quantify prediction accuracy and correlation strength to validate model generalization across chromatographic systems or metabolite sets.
When NOT to use
- Input is a single metabolite or a dataset too small to compute meaningful ranking correlations (ground truth must have at least 3–4 compounds to order).
- Predicted retention orders have not been generated by a retention-order prediction model; use this skill only after model inference, not before.
- Ground-truth retention orders are incomplete, inconsistent, or derived from a different chromatographic instrument/method than the test set (cross-method evaluation requires explicit transfer-learning validation, not standard ranking metrics).
Inputs
- predicted_retention_orders (JSON or tabular format from evaluation_scenarios_main.py output)
- ground_truth_retention_orders (experimental or reference retention-order data)
- evaluation_scenario_configuration (JSON config specifying model, system set, and test split)
Outputs
- ranking_accuracy_scores (per-system or per-fold accuracy metrics)
- spearman_correlation_coefficients (correlation between predicted and true rank orderings)
- results_summary_report (table or HTML report with predicted vs. ground-truth comparisons)
- performance_visualization (optional: plots or figures comparing performance across systems)
How to apply
Execute the repository's evaluation metrics code on the output of the order predictor (RankSVM or SVR). Compare predicted retention-order rankings against ground-truth orderings using ranking accuracy (the fraction of correctly ordered pairs or triplets) and Spearman rank correlation coefficient (which measures monotonic association between predicted and true ranks, robust to individual ranking errors). Generate a results summary table or report listing predicted vs. ground-truth retention orders side-by-side with performance scores for each test system or fold. Document which evaluation scenario was used (baseline_single, all_on_one, etc.) as this affects interpretability—baseline_single reports in-distribution accuracy, while all_on_one evaluates cross-system transfer.
Related tools
- evaluation_scenarios_main.py (Primary experiment script that executes retention-order prediction and computes ranking metrics (accuracy, Spearman correlation) for specified evaluation scenarios) — github.com/aalto-ics-kepaco/retention_order_prediction
- RankSVM (Order predictor model trained on ranking pairs; its predictions are compared against ground truth using ranking metrics) — github.com/aalto-ics-kepaco/retention_order_prediction
- SVR (Alternative order predictor (support vector regression); predictions evaluated using the same ranking metrics as RankSVM) — github.com/aalto-ics-kepaco/retention_order_prediction
- ECCB2018.Rmd (R script for reproducing results summary and visualization; loads and aggregates ranking performance metrics across systems and scenarios) — github.com/aalto-ics-kepaco/retention_order_prediction
Examples
python src/evaluation_scenarios_main.py ranksvm baseline_single 10 -1 results/raw/PredRet/v2/config.json 2 False
Evaluation signals
- Ranking accuracy is reported as a fraction or percentage and falls in the range [0, 1] or [0%, 100%] for each test system.
- Spearman correlation coefficient is in the range [−1, +1]; positive values indicate prediction quality, with |ρ| > 0.7 typically indicating strong correlation.
- Results summary table includes one row per test system or evaluation fold, with columns for predicted order, ground-truth order, accuracy, and Spearman ρ.
- Performance scores are comparable across evaluation scenarios (baseline_single vs. all_on_one) and fingerprint types (MACCS counting vs. binary), enabling systematic model comparison as shown in the paper's tables and figures.
- No NaN, Inf, or missing values in ranking metrics; all test systems are covered and metrics are interpretable.
Limitations
- Ranking metrics (accuracy and Spearman correlation) assume ground-truth orderings are consistent and complete; partial or contradictory orderings (e.g., ties in retention time) may inflate or deflate scores depending on how ties are handled.
- Cross-system evaluation (all_on_one scenario) may show lower performance if the target system was excluded from training (LTSO=False), reflecting domain shift in chromatographic conditions; this is not a model failure but an expected transfer-learning effect.
- Evaluation assumes the test set is independent of training; if data leakage occurs (e.g., same metabolites appear in train and test), metrics will be artificially inflated.
- Spearman correlation is sensitive to outliers in the predicted rankings; a single severely mispredicted order can reduce correlation substantially, especially on small datasets.
Evidence
- [other] Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar).: "Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar)."
- [readme] All experiments of the paper can be reproduced by using the evaluation_scenarios_main.py script with the proper parameters.: "All experiments of the paper can be reproduced by using the evaluation_scenarios_main.py script with the proper parameters."
- [readme] Single system used as training and target: "baseline_single: Single system used as training and target"
- [readme] All systems used for training. Single system used as target. Target system in training (LTSO): True & False: "all_on_one: All systems used for training. Single system used as target. Target system in training (LTSO): True & False"
- [readme] The results will be stored into: results/PredRet/v2/final/ranksvm_slacktype=on_pairs/...: "The results will be stored into: results/PredRet/v2/final/ranksvm_slacktype=on_pairs/.../baseline_single"
1---2name: ranking-performance-evaluation3description: Use when after running retention-order prediction experiments on a test or held-out evaluation dataset.4license: CC-BY-4.05---67# ranking-performance-evaluation89## Summary1011Evaluate retention-order prediction models by comparing predicted rankings against ground-truth orderings using ranking-specific metrics such as ranking accuracy and Spearman correlation. This skill validates whether a learned order predictor (e.g., RankSVM) correctly recovers the relative retention order of metabolites in liquid chromatography.1213## When to use1415Apply this skill after running retention-order prediction experiments on a test or held-out evaluation dataset. Use it when you have both predicted retention orders (from a trained model) and ground-truth retention orders (from experimental or reference data), and you need to quantify prediction accuracy and correlation strength to validate model generalization across chromatographic systems or metabolite sets.1617## When NOT to use1819- Input is a single metabolite or a dataset too small to compute meaningful ranking correlations (ground truth must have at least 3–4 compounds to order).20- Predicted retention orders have not been generated by a retention-order prediction model; use this skill only after model inference, not before.21- Ground-truth retention orders are incomplete, inconsistent, or derived from a different chromatographic instrument/method than the test set (cross-method evaluation requires explicit transfer-learning validation, not standard ranking metrics).2223## Inputs2425- predicted_retention_orders (JSON or tabular format from evaluation_scenarios_main.py output)26- ground_truth_retention_orders (experimental or reference retention-order data)27- evaluation_scenario_configuration (JSON config specifying model, system set, and test split)2829## Outputs3031- ranking_accuracy_scores (per-system or per-fold accuracy metrics)32- spearman_correlation_coefficients (correlation between predicted and true rank orderings)33- results_summary_report (table or HTML report with predicted vs. ground-truth comparisons)34- performance_visualization (optional: plots or figures comparing performance across systems)3536## How to apply3738Execute the repository's evaluation metrics code on the output of the order predictor (RankSVM or SVR). Compare predicted retention-order rankings against ground-truth orderings using ranking accuracy (the fraction of correctly ordered pairs or triplets) and Spearman rank correlation coefficient (which measures monotonic association between predicted and true ranks, robust to individual ranking errors). Generate a results summary table or report listing predicted vs. ground-truth retention orders side-by-side with performance scores for each test system or fold. Document which evaluation scenario was used (baseline_single, all_on_one, etc.) as this affects interpretability—baseline_single reports in-distribution accuracy, while all_on_one evaluates cross-system transfer.3940## Related tools4142- **evaluation_scenarios_main.py** (Primary experiment script that executes retention-order prediction and computes ranking metrics (accuracy, Spearman correlation) for specified evaluation scenarios) — github.com/aalto-ics-kepaco/retention_order_prediction43- **RankSVM** (Order predictor model trained on ranking pairs; its predictions are compared against ground truth using ranking metrics) — github.com/aalto-ics-kepaco/retention_order_prediction44- **SVR** (Alternative order predictor (support vector regression); predictions evaluated using the same ranking metrics as RankSVM) — github.com/aalto-ics-kepaco/retention_order_prediction45- **ECCB2018.Rmd** (R script for reproducing results summary and visualization; loads and aggregates ranking performance metrics across systems and scenarios) — github.com/aalto-ics-kepaco/retention_order_prediction4647## Examples4849```50python src/evaluation_scenarios_main.py ranksvm baseline_single 10 -1 results/raw/PredRet/v2/config.json 2 False51```5253## Evaluation signals5455- Ranking accuracy is reported as a fraction or percentage and falls in the range [0, 1] or [0%, 100%] for each test system.56- Spearman correlation coefficient is in the range [−1, +1]; positive values indicate prediction quality, with |ρ| > 0.7 typically indicating strong correlation.57- Results summary table includes one row per test system or evaluation fold, with columns for predicted order, ground-truth order, accuracy, and Spearman ρ.58- Performance scores are comparable across evaluation scenarios (baseline_single vs. all_on_one) and fingerprint types (MACCS counting vs. binary), enabling systematic model comparison as shown in the paper's tables and figures.59- No NaN, Inf, or missing values in ranking metrics; all test systems are covered and metrics are interpretable.6061## Limitations6263- Ranking metrics (accuracy and Spearman correlation) assume ground-truth orderings are consistent and complete; partial or contradictory orderings (e.g., ties in retention time) may inflate or deflate scores depending on how ties are handled.64- Cross-system evaluation (all_on_one scenario) may show lower performance if the target system was excluded from training (LTSO=False), reflecting domain shift in chromatographic conditions; this is not a model failure but an expected transfer-learning effect.65- Evaluation assumes the test set is independent of training; if data leakage occurs (e.g., same metabolites appear in train and test), metrics will be artificially inflated.66- Spearman correlation is sensitive to outliers in the predicted rankings; a single severely mispredicted order can reduce correlation substantially, especially on small datasets.6768## Evidence6970- [other] Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar).: "Capture predicted retention orders and compare against ground-truth orderings using the repository's evaluation metrics (ranking accuracy, Spearman correlation, or similar)."71- [readme] All experiments of the paper can be reproduced by using the evaluation_scenarios_main.py script with the proper parameters.: "All experiments of the paper can be reproduced by using the evaluation_scenarios_main.py script with the proper parameters."72- [readme] Single system used as training and target: "baseline_single: Single system used as training and target"73- [readme] All systems used for training. Single system used as target. Target system in training (LTSO): True & False: "all_on_one: All systems used for training. Single system used as target. Target system in training (LTSO): True & False"74- [readme] The results will be stored into: results/PredRet/v2/final/ranksvm_slacktype=on_pairs/...: "The results will be stored into: results/PredRet/v2/final/ranksvm_slacktype=on_pairs/.../baseline_single"
HolobiomicsLab