quality-control-threshold-application
License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution.
Summary
Apply a quantitative missingness threshold to filter features (metabolites) from high-dimensional replicate experiments, retaining only those meeting a minimum presence criterion across samples. This skill removes low-quality features before downstream analysis and is essential for preparing metabolomics data for reproducibility assessment.
When to use
When you have a raw feature abundance matrix with missing values (NAs or zeros) and need to remove features with excessive sparsity before imputation or reproducibility analysis. Apply this skill when the proportion of missing data per feature varies substantially and a biological or technical minimum presence threshold is justified (e.g., a metabolite must be quantifiable in at least 20% of replicates to be retained for analysis).
When NOT to use
- Input data already consists of a pre-filtered feature table with documented minimum presence thresholds applied.
- Missingness is not random or informative; features with high missingness are biologically relevant (e.g., conditionally expressed markers) and removal would bias results.
- Sample size is very small (N < 5) such that a 20% presence threshold cannot be met for any feature; choose a lower threshold or alternative strategy.
Inputs
- Raw feature abundance matrix (e.g., SummarizedExperiment object with metabolite rows × sample columns)
- Numeric threshold for maximum acceptable missingness as a proportion (e.g., 0.80 for 80%)
- Missing value convention (NA, zero, or software-specific representation)
Outputs
- Filtered feature abundance matrix with features below missingness threshold retained
- Feature count before and after filtering (e.g., 662 → 645 metabolites)
- List of removed feature identifiers and their individual missingness proportions
How to apply
Calculate the proportion of missing values (NA or zero, following MSPrep convention) for each feature across all samples. Define a missingness threshold (e.g., 80%) and retain only features with missingness at or below that threshold, equivalently ensuring each feature is present in at least the complementary percentage of samples (e.g., ≥20% presence in at least 4 of 20 samples). Remove features exceeding the threshold and output the filtered abundance matrix with feature identifiers and sample annotations preserved. The rationale is to exclude metabolites too sparsely measured to provide reliable signal in downstream reproducibility or statistical analyses, while minimizing feature loss (typically 2–5% of features).
Related tools
- MSPrep (Pre-processing software that establishes the missing-value convention (NA/zero) and filtering methodology applied to raw metabolomics data)
- marr (R/Bioconductor package containing the msprepCOPD dataset and downstream reproducibility assessment; filtering is a preprocessing step before marr reproducibility calculations) — https://github.com/Ghoshlab/marr
- R (Programming language used to load SummarizedExperiment objects, compute missingness proportions, and apply filter logic)
- Bioconductor (Framework providing SummarizedExperiment data structures and standard methods for manipulating high-dimensional genomics/metabolomics assay data)
Examples
# Load msprepCOPD from marr package; calculate missingness per metabolite; retain metabolites with ≤80% missingness
library(marr); data(msprepCOPD); miss_prop <- rowMeans(is.na(assay(msprepCOPD))); filtered <- msprepCOPD[miss_prop <= 0.80, ]; cat('Retained:', nrow(filtered), 'metabolites\n')
Evaluation signals
- Feature count reduction is within expected range (2–5% loss typical for 80% threshold) and no features with ≥20% presence are incorrectly removed.
- Missingness proportion for every retained feature is ≤80% (or ≤ specified threshold); every removed feature has missingness >80%.
- Output matrix dimensions match expected filtered counts (e.g., 645 features × 20 samples) and all feature identifiers and sample annotations are preserved.
- No numerical or data-type errors during missingness calculation (check for proper handling of NA, zero, and mixed encodings across software).
- Downstream reproducibility or imputation analysis proceeds without failures, indicating data quality is sufficient for next step.
Limitations
- Threshold choice (e.g., 80%) is somewhat arbitrary; sensitivity analysis on nearby thresholds (70%–90%) may reveal whether conclusions are robust to threshold variation.
- Applying a uniform threshold across all features may be suboptimal if different feature classes (e.g., lipids vs. amino acids) have intrinsically different missingness rates; stratified or adaptive thresholds are not addressed in this article.
- Missingness filtering does not account for the magnitude or quality of non-missing values; a feature present in 25% of samples may still have poor signal-to-noise or low dynamic range in those samples.
Evidence
- [intro] The MSPrep filtering step removes metabolites exceeding 80% missingness, reducing the raw metabolite count from 662 to 645 (a loss of 17 metabolites or 2.6% of features).: "The MSPrep filtering step removes metabolites exceeding 80% missingness, reducing the raw metabolite count from 662 to 645 (a loss of 17 metabolites or 2.6% of features)."
- [intro] Filtering workflow specifies calculation and retention logic: identify and retain only metabolites with ≤80% missingness (i.e., present in ≥20% of samples, or equivalently ≥4 of 20 samples).: "Identify and retain only metabolites with ≤80% missingness (i.e., present in ≥20% of samples, or equivalently ≥4 of 20 samples)."
- [intro] Workflow step description from article enrichment: 'Filtering: Metabolites are removed if they are missing more than 80% of the samples': "Filtering: Metabolites are removed if they are missing more than 80% of the samples"
- [readme] The marr package README confirms the msprepCOPD dataset contains pre-processed data after filtering to 645 metabolites.: "The
marr package contains a pre-processed data SummarizedExperiment assay object of 645 metabolites (features) measured in plasma and 20 biological replicates"
1---2name: quality-control-threshold-application-23description: Use when when you have a raw feature abundance matrix with missing values (NAs or zeros) and need to remove features with excessive sparsity before imputation or reproducibility analysis.4license: CC-BY-4.05---67# quality-control-threshold-application89> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->10## Summary1112Apply a quantitative missingness threshold to filter features (metabolites) from high-dimensional replicate experiments, retaining only those meeting a minimum presence criterion across samples. This skill removes low-quality features before downstream analysis and is essential for preparing metabolomics data for reproducibility assessment.1314## When to use1516When you have a raw feature abundance matrix with missing values (NAs or zeros) and need to remove features with excessive sparsity before imputation or reproducibility analysis. Apply this skill when the proportion of missing data per feature varies substantially and a biological or technical minimum presence threshold is justified (e.g., a metabolite must be quantifiable in at least 20% of replicates to be retained for analysis).1718## When NOT to use1920- Input data already consists of a pre-filtered feature table with documented minimum presence thresholds applied.21- Missingness is not random or informative; features with high missingness are biologically relevant (e.g., conditionally expressed markers) and removal would bias results.22- Sample size is very small (N < 5) such that a 20% presence threshold cannot be met for any feature; choose a lower threshold or alternative strategy.2324## Inputs2526- Raw feature abundance matrix (e.g., SummarizedExperiment object with metabolite rows × sample columns)27- Numeric threshold for maximum acceptable missingness as a proportion (e.g., 0.80 for 80%)28- Missing value convention (NA, zero, or software-specific representation)2930## Outputs3132- Filtered feature abundance matrix with features below missingness threshold retained33- Feature count before and after filtering (e.g., 662 → 645 metabolites)34- List of removed feature identifiers and their individual missingness proportions3536## How to apply3738Calculate the proportion of missing values (NA or zero, following MSPrep convention) for each feature across all samples. Define a missingness threshold (e.g., 80%) and retain only features with missingness at or below that threshold, equivalently ensuring each feature is present in at least the complementary percentage of samples (e.g., ≥20% presence in at least 4 of 20 samples). Remove features exceeding the threshold and output the filtered abundance matrix with feature identifiers and sample annotations preserved. The rationale is to exclude metabolites too sparsely measured to provide reliable signal in downstream reproducibility or statistical analyses, while minimizing feature loss (typically 2–5% of features).3940## Related tools4142- **MSPrep** (Pre-processing software that establishes the missing-value convention (NA/zero) and filtering methodology applied to raw metabolomics data)43- **marr** (R/Bioconductor package containing the msprepCOPD dataset and downstream reproducibility assessment; filtering is a preprocessing step before marr reproducibility calculations) — https://github.com/Ghoshlab/marr44- **R** (Programming language used to load SummarizedExperiment objects, compute missingness proportions, and apply filter logic)45- **Bioconductor** (Framework providing SummarizedExperiment data structures and standard methods for manipulating high-dimensional genomics/metabolomics assay data)4647## Examples4849```50# Load msprepCOPD from marr package; calculate missingness per metabolite; retain metabolites with ≤80% missingness51library(marr); data(msprepCOPD); miss_prop <- rowMeans(is.na(assay(msprepCOPD))); filtered <- msprepCOPD[miss_prop <= 0.80, ]; cat('Retained:', nrow(filtered), 'metabolites\n')52```5354## Evaluation signals5556- Feature count reduction is within expected range (2–5% loss typical for 80% threshold) and no features with ≥20% presence are incorrectly removed.57- Missingness proportion for every retained feature is ≤80% (or ≤ specified threshold); every removed feature has missingness >80%.58- Output matrix dimensions match expected filtered counts (e.g., 645 features × 20 samples) and all feature identifiers and sample annotations are preserved.59- No numerical or data-type errors during missingness calculation (check for proper handling of NA, zero, and mixed encodings across software).60- Downstream reproducibility or imputation analysis proceeds without failures, indicating data quality is sufficient for next step.6162## Limitations6364- Threshold choice (e.g., 80%) is somewhat arbitrary; sensitivity analysis on nearby thresholds (70%–90%) may reveal whether conclusions are robust to threshold variation.65- Applying a uniform threshold across all features may be suboptimal if different feature classes (e.g., lipids vs. amino acids) have intrinsically different missingness rates; stratified or adaptive thresholds are not addressed in this article.66- Missingness filtering does not account for the magnitude or quality of non-missing values; a feature present in 25% of samples may still have poor signal-to-noise or low dynamic range in those samples.6768## Evidence6970- [intro] The MSPrep filtering step removes metabolites exceeding 80% missingness, reducing the raw metabolite count from 662 to 645 (a loss of 17 metabolites or 2.6% of features).: "The MSPrep filtering step removes metabolites exceeding 80% missingness, reducing the raw metabolite count from 662 to 645 (a loss of 17 metabolites or 2.6% of features)."71- [intro] Filtering workflow specifies calculation and retention logic: identify and retain only metabolites with ≤80% missingness (i.e., present in ≥20% of samples, or equivalently ≥4 of 20 samples).: "Identify and retain only metabolites with ≤80% missingness (i.e., present in ≥20% of samples, or equivalently ≥4 of 20 samples)."72- [intro] Workflow step description from article enrichment: 'Filtering: Metabolites are removed if they are missing more than 80% of the samples': "Filtering: Metabolites are removed if they are missing more than 80% of the samples"73- [readme] The marr package README confirms the msprepCOPD dataset contains pre-processed data after filtering to 645 metabolites.: "The `marr` package contains a pre-processed data `SummarizedExperiment` assay object of 645 metabolites (features) measured in plasma and 20 biological replicates"
HolobiomicsLab 