robustness-metric-computation-metabolomics
License: noncommercial — confirm your use is a permitted (noncommercial) purpose before applying; commercial use requires a separate license (see metadata.tool_license).
Summary
Compute and compare robustness metrics (bias, variance, recovery of ground truth) and computational speed across multiple normalization methods applied to metabolomics data. This skill enables quantitative evaluation of normalization method performance for reproducibility assessment and method recommendation.
When to use
When you have applied multiple normalization methods (e.g., tGAM, rGAM, rLOESS, QC-RLSC, QC-RSC) to metabolomics datasets and need to rank them by robustness and execution speed to determine which method is most suitable for your experimental design and computational constraints.
When NOT to use
- Input data lacks batch structure or QC sample replicates — robustness metrics require within-batch and cross-batch variation to be meaningful.
- No ground truth reference is available and the biological sample set is too small to estimate true signal reliably — bias and recovery metrics will be unreliable.
- Only one normalization method has been applied — comparative ranking requires at least two methods to be evaluated side-by-side.
Inputs
- Unnormalized metabolomics intensity matrix (rows = compounds, columns = samples)
- Batch assignment vector (one value per sample)
- Sample type vector indicating QC vs. biological samples
- Ground truth or spike-in reference values (optional but recommended for bias/recovery calculation)
Outputs
- Robustness metrics table (columns = methods; rows = bias, variance, recovery %; units specified)
- Execution time vector (one value per method, in seconds or milliseconds)
- Comparative ranking plots (e.g., box plots of bias/variance by method, scatter of speed vs. robustness)
- Ranked method list (sorted by robustness score or by user-specified priority order)
How to apply
Run each normalization method on a standardized metabolomics dataset with known ground truth or spike-in controls, recording both the normalized intensity matrix and execution time. Calculate bias as deviation from ground truth (e.g., mean relative error), variance as stability across replicates or batches, and recovery rate as the percentage of true signal retained. Normalize metrics to a common scale (e.g., 0–100 or standardized units) and generate a comparative ranking table and visualization (e.g., box plots or radar charts) matching the published paper's comparative evaluation structure. Verify that reproduced metric ranges and rankings align with the paper's reported findings within acceptable tolerance (typically ±10% relative difference for bias and variance, ±5% for execution time).
Related tools
- Metanorm (R package implementing five normalization methods (tGAM, rGAM, rLOESS, QC-RLSC, QC-RSC) on which robustness metrics are computed) — https://github.com/UGent-LIMET/Metanorm
- R (Statistical environment for running Metanorm, computing robustness metrics (bias, variance, recovery), and generating comparative visualizations)
Examples
normdat_tGAM <- metanorm(rawdata[1:5,], model="tGAM", type=metanorm.qc, QCcheck=TRUE, batch=batch); normdat_QC <- metanorm(rawdata[1:5,], model="QC-RLSC", type=metanorm.qc, QConly=TRUE, batch=batch); # Then compute bias, variance, and execution times, and compare rankings.
Evaluation signals
- Reproduced metric values (bias, variance, recovery %) for each method fall within ±10% relative difference of the published paper's reported ranges.
- Ranking order of methods by robustness matches the paper's findings (e.g., tGAM ranked highest for robustness, rGAM and rLOESS ranked higher for speed).
- Execution time measurements are internally consistent across repeated runs and scale reasonably with dataset size (compound count and sample count).
- Comparative visualization (plots and tables) shows clear separation between methods on at least one robustness or speed dimension, demonstrating discriminative power.
- QC vs. biological sample discrepancy checks (if applicable) flag or report any samples with unexpected normalization behavior, confirming method reliability assessment.
Limitations
- Robustness metrics depend critically on the representativeness and abundance of QC samples; sparse or biased QC sampling may yield unreliable comparisons.
- Ground truth recovery cannot be computed without spike-in controls or independent reference measurements; bias and recovery estimates will be approximate if based on replicate variance alone.
- Execution speed is platform- and dataset-dependent; reported timings may not generalize to smaller or larger metabolomics cohorts or different computational hardware.
- The paper recommends using both QC and biological samples for normalization, but only QC-only methods (e.g., QC-RLSC with QConly=TRUE) may show different robustness profiles; ensure parameter consistency across comparisons.
Evidence
- [other] Comparative evaluation workflow: "Compute robustness metrics (e.g., bias, variance, recovery of ground truth) and execution time for each method."
- [readme] Five methods evaluated: "The R package implements three (new) robust normalization methods (tGAM, rGAM and rLOESS), alongside formerly proposed ones (QC-RLSC, QC-RSC)."
- [readme] Recommended method ranking by robustness: "tGAM is recommended due to its superior robustness, but rGAM and rLOESS are faster."
- [other] Ranking verification criterion: "Verify that reproduced rankings and metric ranges align with published findings within acceptable tolerance."
- [readme] QC and biological sample recommendation: "We further recommend using both QC as well as biological samples for normalization, and to have metanorm check for discrepancies between QC and biological samples."
1---2name: robustness-metric-computation-metabolomics3description: Use when when you have applied multiple normalization methods (e.g., tGAM, rGAM, rLOESS, QC-RLSC, QC-RSC) to metabolomics datasets and need to rank them by robustness and execution speed to determine which method is most suitable for your experimental design and computational constraints.4license: CC-BY-4.05---67# robustness-metric-computation-metabolomics89> **License: noncommercial** — confirm your use is a permitted (noncommercial) purpose before applying; commercial use requires a separate license (see `metadata.tool_license`). <!-- asb-license-banner -->10## Summary1112Compute and compare robustness metrics (bias, variance, recovery of ground truth) and computational speed across multiple normalization methods applied to metabolomics data. This skill enables quantitative evaluation of normalization method performance for reproducibility assessment and method recommendation.1314## When to use1516When you have applied multiple normalization methods (e.g., tGAM, rGAM, rLOESS, QC-RLSC, QC-RSC) to metabolomics datasets and need to rank them by robustness and execution speed to determine which method is most suitable for your experimental design and computational constraints.1718## When NOT to use1920- Input data lacks batch structure or QC sample replicates — robustness metrics require within-batch and cross-batch variation to be meaningful.21- No ground truth reference is available and the biological sample set is too small to estimate true signal reliably — bias and recovery metrics will be unreliable.22- Only one normalization method has been applied — comparative ranking requires at least two methods to be evaluated side-by-side.2324## Inputs2526- Unnormalized metabolomics intensity matrix (rows = compounds, columns = samples)27- Batch assignment vector (one value per sample)28- Sample type vector indicating QC vs. biological samples29- Ground truth or spike-in reference values (optional but recommended for bias/recovery calculation)3031## Outputs3233- Robustness metrics table (columns = methods; rows = bias, variance, recovery %; units specified)34- Execution time vector (one value per method, in seconds or milliseconds)35- Comparative ranking plots (e.g., box plots of bias/variance by method, scatter of speed vs. robustness)36- Ranked method list (sorted by robustness score or by user-specified priority order)3738## How to apply3940Run each normalization method on a standardized metabolomics dataset with known ground truth or spike-in controls, recording both the normalized intensity matrix and execution time. Calculate bias as deviation from ground truth (e.g., mean relative error), variance as stability across replicates or batches, and recovery rate as the percentage of true signal retained. Normalize metrics to a common scale (e.g., 0–100 or standardized units) and generate a comparative ranking table and visualization (e.g., box plots or radar charts) matching the published paper's comparative evaluation structure. Verify that reproduced metric ranges and rankings align with the paper's reported findings within acceptable tolerance (typically ±10% relative difference for bias and variance, ±5% for execution time).4142## Related tools4344- **Metanorm** (R package implementing five normalization methods (tGAM, rGAM, rLOESS, QC-RLSC, QC-RSC) on which robustness metrics are computed) — https://github.com/UGent-LIMET/Metanorm45- **R** (Statistical environment for running Metanorm, computing robustness metrics (bias, variance, recovery), and generating comparative visualizations)4647## Examples4849```50normdat_tGAM <- metanorm(rawdata[1:5,], model="tGAM", type=metanorm.qc, QCcheck=TRUE, batch=batch); normdat_QC <- metanorm(rawdata[1:5,], model="QC-RLSC", type=metanorm.qc, QConly=TRUE, batch=batch); # Then compute bias, variance, and execution times, and compare rankings.51```5253## Evaluation signals5455- Reproduced metric values (bias, variance, recovery %) for each method fall within ±10% relative difference of the published paper's reported ranges.56- Ranking order of methods by robustness matches the paper's findings (e.g., tGAM ranked highest for robustness, rGAM and rLOESS ranked higher for speed).57- Execution time measurements are internally consistent across repeated runs and scale reasonably with dataset size (compound count and sample count).58- Comparative visualization (plots and tables) shows clear separation between methods on at least one robustness or speed dimension, demonstrating discriminative power.59- QC vs. biological sample discrepancy checks (if applicable) flag or report any samples with unexpected normalization behavior, confirming method reliability assessment.6061## Limitations6263- Robustness metrics depend critically on the representativeness and abundance of QC samples; sparse or biased QC sampling may yield unreliable comparisons.64- Ground truth recovery cannot be computed without spike-in controls or independent reference measurements; bias and recovery estimates will be approximate if based on replicate variance alone.65- Execution speed is platform- and dataset-dependent; reported timings may not generalize to smaller or larger metabolomics cohorts or different computational hardware.66- The paper recommends using both QC and biological samples for normalization, but only QC-only methods (e.g., QC-RLSC with QConly=TRUE) may show different robustness profiles; ensure parameter consistency across comparisons.6768## Evidence6970- [other] Comparative evaluation workflow: "Compute robustness metrics (e.g., bias, variance, recovery of ground truth) and execution time for each method."71- [readme] Five methods evaluated: "The R package implements three (new) robust normalization methods (tGAM, rGAM and rLOESS), alongside formerly proposed ones (QC-RLSC, QC-RSC)."72- [readme] Recommended method ranking by robustness: "tGAM is recommended due to its superior robustness, but rGAM and rLOESS are faster."73- [other] Ranking verification criterion: "Verify that reproduced rankings and metric ranges align with published findings within acceptable tolerance."74- [readme] QC and biological sample recommendation: "We further recommend using both QC as well as biological samples for normalization, and to have metanorm check for discrepancies between QC and biological samples."
