regression-performance-metric-computation
Summary
Compute standard regression performance metrics (mean absolute error, R², and other statistics) on model predictions against ground-truth labels to quantify generalizability of graph neural network collision cross section predictions. This skill validates whether a trained model reproduces reported performance benchmarks.
When to use
After generating collision cross section predictions on a validation or test set using a trained graph neural network model, and you need to quantify prediction accuracy and compare against reported performance metrics in the literature or prior experimental runs.
When NOT to use
- Predictions have not yet been generated on the test set—first run the model inference step.
- Test set labels are missing or misaligned with predictions (shape or index mismatch).
- You are evaluating classification performance rather than regression—use classification metrics (accuracy, F1, ROC-AUC) instead.
Inputs
- predicted_ccs_values (numeric array or column from predictions file)
- ground_truth_ccs_values (numeric column from test/validation dataset parquet file)
- test_set_metadata (dataset identifiers, e.g., 'ccsbase_3d.parquet', 'metlin_train_3d.parquet')
Outputs
- metrics_file (JSON or CSV with MAE, R², and other regression statistics)
- performance_summary (structured record of metric names and values for comparison against reported benchmarks)
How to apply
Load the model's predicted collision cross section values and the corresponding ground-truth CCS values from the test or validation dataset (typically stored in parquet format with columns like 'ccs' for ground truth and prediction output). Compute mean absolute error (MAE), R² (coefficient of determination), and any other regression statistics reported in the paper. Save the computed metrics to a structured output file (e.g., JSON or CSV) that records the metric names, values, and optionally dataset split metadata (train set, test set). The rationale is that regression metrics on held-out data directly measure model generalizability across different CCS databases (e.g., training on METLIN and testing on CCSBase reveals cross-database performance).
Related tools
Examples
poetry run python scripts/train-test.py --prefix "train-metlin-test-ccsbase" --train-input-file "ccs-prediction/metlin_train_3d.parquet" --test-input-file "ccs-prediction/ccsbase_3d.parquet" --parameter-path "parameter/parameter-train-metlin-test-metlin.json" --model-output-file "model/train-metlin-test-metlin.h5" --ccs-column-name "ccs" --dropout-rate 0.1 --epochs 400
Evaluation signals
- Computed metrics match or closely reproduce the values reported in Engler et al. (2024) for each train/test dataset pair (e.g., 'train-metlin-test-ccsbase').
- Metrics file contains all expected fields: metric name (e.g., 'MAE', 'R2'), numeric value, and dataset split identifiers.
- R² is in range [−∞, 1] and MAE is non-negative; for good models on CCS prediction, R² should be > 0.8 and MAE should be < ~5 Ų (depending on database and adduct).
- Predictions and ground-truth arrays have matching lengths and no NaN/inf values in metric computation.
- Cross-database generalization comparison: metrics for 'train-metlin-test-ccsbase' should be interpretable as a measure of domain shift (expected to be worse than same-database splits).
Limitations
- Metrics are sensitive to data preprocessing and normalization; ensure training and test sets are processed identically (coordinates present/absent, SMILES canonicalization, adduct encoding).
- R² can be misleading if the test set distribution differs significantly from training (cross-database evaluation may show lower R² even if model is sound).
- No single metric fully captures generalizability; MAE and R² should be reported together with residual analysis and per-adduct or per-molecular-weight-bin breakdowns (see reproduce_figures notebooks).
- Pre-trained model weights and hyperparameters (dropout=0.1, epochs=400) must match those used to generate reported metrics; using different hyperparameters will produce non-comparable results.
Evidence
- [other] Compute reported performance metrics (e.g., mean absolute error, R², or other regression statistics) and save results to a metrics file.: "Compute reported performance metrics (e.g., mean absolute error, R², or other regression statistics) and save results to a metrics file."
- [readme] Predictions are available and can be directly downloaded from [...] The files should be unzipped and placed in the
data directory.: "Predictions are available and can be directly downloaded from . The files should be unzipped"
- [readme] Run the notebooks located in the
notebooks corresponding to each analysis. [...] reproduce_figures: the name of the notebooks indicates which notebook can reproduce which figures of the manuscript: "Run the notebooks located in the notebooks corresponding to each analysis. [...] reproduce_figures: the name of the notebooks indicates which notebook can reproduce which figures"
- [readme] prefix is used to generate the output files of the predictions of the test set: "prefix is used to generate the output files of the predictions of the test set"
1---2name: regression-performance-metric-computation3description: Use when after generating collision cross section predictions on a validation or test set using a trained graph neural network model, and you need to quantify prediction accuracy and compare against reported performance metrics in the literature or prior experimental runs.4license: CC-BY-4.05---67# regression-performance-metric-computation89## Summary1011Compute standard regression performance metrics (mean absolute error, R², and other statistics) on model predictions against ground-truth labels to quantify generalizability of graph neural network collision cross section predictions. This skill validates whether a trained model reproduces reported performance benchmarks.1213## When to use1415After generating collision cross section predictions on a validation or test set using a trained graph neural network model, and you need to quantify prediction accuracy and compare against reported performance metrics in the literature or prior experimental runs.1617## When NOT to use1819- Predictions have not yet been generated on the test set—first run the model inference step.20- Test set labels are missing or misaligned with predictions (shape or index mismatch).21- You are evaluating classification performance rather than regression—use classification metrics (accuracy, F1, ROC-AUC) instead.2223## Inputs2425- predicted_ccs_values (numeric array or column from predictions file)26- ground_truth_ccs_values (numeric column from test/validation dataset parquet file)27- test_set_metadata (dataset identifiers, e.g., 'ccsbase_3d.parquet', 'metlin_train_3d.parquet')2829## Outputs3031- metrics_file (JSON or CSV with MAE, R², and other regression statistics)32- performance_summary (structured record of metric names and values for comparison against reported benchmarks)3334## How to apply3536Load the model's predicted collision cross section values and the corresponding ground-truth CCS values from the test or validation dataset (typically stored in parquet format with columns like 'ccs' for ground truth and prediction output). Compute mean absolute error (MAE), R² (coefficient of determination), and any other regression statistics reported in the paper. Save the computed metrics to a structured output file (e.g., JSON or CSV) that records the metric names, values, and optionally dataset split metadata (train set, test set). The rationale is that regression metrics on held-out data directly measure model generalizability across different CCS databases (e.g., training on METLIN and testing on CCSBase reveals cross-database performance).3738## Related tools3940- **train-test.py** (Generates predictions and can optionally compute metrics; orchestrates model evaluation workflow) — https://github.com/enveda/ccs-prediction41- **sklearn.metrics** (Python library providing mean_absolute_error, r2_score, and other regression metric functions)42- **reproduce_figures notebooks** (Jupyter notebooks that compute and visualize performance metrics for manuscript figures) — https://github.com/enveda/ccs-prediction/tree/main/notebooks/reproduce_figures4344## Examples4546```47poetry run python scripts/train-test.py --prefix "train-metlin-test-ccsbase" --train-input-file "ccs-prediction/metlin_train_3d.parquet" --test-input-file "ccs-prediction/ccsbase_3d.parquet" --parameter-path "parameter/parameter-train-metlin-test-metlin.json" --model-output-file "model/train-metlin-test-metlin.h5" --ccs-column-name "ccs" --dropout-rate 0.1 --epochs 40048```4950## Evaluation signals5152- Computed metrics match or closely reproduce the values reported in Engler et al. (2024) for each train/test dataset pair (e.g., 'train-metlin-test-ccsbase').53- Metrics file contains all expected fields: metric name (e.g., 'MAE', 'R2'), numeric value, and dataset split identifiers.54- R² is in range [−∞, 1] and MAE is non-negative; for good models on CCS prediction, R² should be > 0.8 and MAE should be < ~5 Ų (depending on database and adduct).55- Predictions and ground-truth arrays have matching lengths and no NaN/inf values in metric computation.56- Cross-database generalization comparison: metrics for 'train-metlin-test-ccsbase' should be interpretable as a measure of domain shift (expected to be worse than same-database splits).5758## Limitations5960- Metrics are sensitive to data preprocessing and normalization; ensure training and test sets are processed identically (coordinates present/absent, SMILES canonicalization, adduct encoding).61- R² can be misleading if the test set distribution differs significantly from training (cross-database evaluation may show lower R² even if model is sound).62- No single metric fully captures generalizability; MAE and R² should be reported together with residual analysis and per-adduct or per-molecular-weight-bin breakdowns (see reproduce_figures notebooks).63- Pre-trained model weights and hyperparameters (dropout=0.1, epochs=400) must match those used to generate reported metrics; using different hyperparameters will produce non-comparable results.6465## Evidence6667- [other] Compute reported performance metrics (e.g., mean absolute error, R², or other regression statistics) and save results to a metrics file.: "Compute reported performance metrics (e.g., mean absolute error, R², or other regression statistics) and save results to a metrics file."68- [readme] Predictions are available and can be directly downloaded from [...] The files should be unzipped and placed in the `data` directory.: "Predictions are available and can be directly downloaded from [](https://doi.org/10.5281/zenodo.11199061). The files should be unzipped"69- [readme] Run the notebooks located in the `notebooks` corresponding to each analysis. [...] reproduce_figures: the name of the notebooks indicates which notebook can reproduce which figures of the manuscript: "Run the notebooks located in the `notebooks` corresponding to each analysis. [...] reproduce_figures: the name of the notebooks indicates which notebook can reproduce which figures"70- [readme] prefix is used to generate the output files of the predictions of the test set: "prefix is used to generate the output files of the predictions of the test set"
HolobiomicsLab