quantification-table-normalization
Summary
Transform and normalize quantification tables (metabolite or feature abundance matrices) into the internal data format expected by msFeaST for downstream feature selection and statistical analysis. This preprocessing step ensures compatibility with the pipeline's statistical tests and interactive visualization.
When to use
You have raw quantification data (abundance or intensity values across samples and features) from mass spectrometry or similar high-dimensional assays and need to prepare it for msFeaST's feature selection workflow, which requires standardized internal data structures compatible with R-based statistical tests (e.g., globaltest).
When NOT to use
- Quantification data is already in msFeaST's internal format or has been validated and formatted by a prior preprocessing run.
- You only need to inspect pre-computed results without re-processing raw abundance data—use the dashboard directly with existing dashboard_data.json.
- Your quantification matrix lacks metadata or spectral data mappings, making alignment impossible.
Inputs
- quantification table (CSV, TSV, or pandas DataFrame with samples × features abundance matrix)
- metadata table (for sample ID validation and alignment)
- data filepath arguments (user-specified locations of quantification files)
Outputs
- normalized quantification data structure (Python dictionary or DataFrame in msFeaST internal format)
- validated sample-feature mappings ready for pipeline input
- intermediate data object compatible with R globaltest and statistical functions
How to apply
Load the quantification table (typically a pandas DataFrame or CSV with samples as rows and features/metabolites as columns) using the msFeaST preprocessing notebook (preprocessing_mushroom_type_comparison.ipynb). Parse and validate the input data structure to ensure it meets msFeaST requirements—notably sample identifiers that map to metadata rows and feature identifiers that correspond to spectral data. Transform the quantification table into the internal Python data structure (dictionary or DataFrame) expected by msFeaST, which aligns with downstream R statistical dependencies including the globaltest package. The normalization step may include handling missing values, scaling, or log-transformation depending on the assay type; refer to the notebook's magenta-italicized user input sections for dataset-specific parameters. Validate alignment between quantification sample IDs and metadata table rows before proceeding to the pipeline.
Related tools
- msFeaST (provides the preprocessing notebook, internal data format specification, and statistical pipeline that consumes the normalized quantification table) — https://github.com/kevinmildau/msFeaST
- jupyter-notebook (execution environment for the preprocessing_mushroom_type_comparison.ipynb notebook that loads, parses, and normalizes the quantification table)
- pandas (used implicitly within msFeaST preprocessing to load and manipulate quantification DataFrames)
- globaltest (R package) (downstream R dependency that receives the normalized quantification data for feature selection and statistical testing)
Examples
jupyter-notebook preprocessing_mushroom_type_comparison.ipynb # then modify data filepath arguments to your quantification, metadata, and spectral data files and execute cells
Evaluation signals
- Quantification table sample IDs match all rows in the metadata table with no missing or orphaned mappings.
- Feature identifiers in the quantification table align with feature IDs in the spectral data without duplicates or conflicts.
- The normalized data structure can be serialized and passed to the msFeaST pipeline without schema or type errors.
- R globaltest and associated statistical functions (dplyr, tibble) execute without error on the normalized quantification object.
- The pipeline produces a valid JSON output file that loads successfully in the msFeaST interactive dashboard.
Limitations
- msFeaST preprocessing and pipeline workflow have been tested on macOS and Linux; Windows support is currently being worked on.
- R package installation via conda is not reliable for the required packages; remotes and BiocManager must be used to control R package versions, which may require several minutes for compilation.
- The skill assumes that sample identifiers and feature identifiers are stable and unambiguous across the quantification, metadata, and spectral data files; user must verify alignment before pipeline execution.
- No changelog is available to track changes in data format specifications between msFeaST versions, potentially causing compatibility issues if updating the tool.
Evidence
- [other] Transform and normalize the quantification and metadata tables into the internal data format expected by msFeaST.: "Transform and normalize the quantification and metadata tables into the internal data format expected by msFeaST"
- [other] Parse and validate the input data structures to ensure compatibility with msFeaST requirements.: "Parse and validate the input data structures to ensure compatibility with msFeaST requirements"
- [readme] Navigate to the notebook folder and open the preprocessing_mushroom_type_comparison.ipynb and msfeast_pipeline_mushroom_type_comparison.ipynb notebooks on your local machine. These notebooks contain a complete example of quantification table, metadata table, and spectral data processing required for msFeaST, as well as a complete use-case example.: "These notebooks contain a complete example of quantification table, metadata table, and spectral data processing required for msFeaST"
- [readme] Text in magenta italics font highlights required user input for the pipeline.: "Text in magenta italics font highlights required user input for the pipeline"
- [readme] The current msFeaST pre-processing and pipeline workflow has been tested on macos and should work identically on linux operating systems. Windows support is currently being worked on.: "The current msFeaST pre-processing and pipeline workflow has been tested on macos and should work identically on linux operating systems. Windows support is currently being worked on"
1---2name: quantification-table-normalization3description: Use when you have raw quantification data (abundance or intensity values across samples and features) from mass spectrometry or similar high-dimensional assays and need to prepare it for msFeaST's feature selection workflow, which requires standardized internal data structures compatible with.4license: CC-BY-4.05---67# quantification-table-normalization89## Summary1011Transform and normalize quantification tables (metabolite or feature abundance matrices) into the internal data format expected by msFeaST for downstream feature selection and statistical analysis. This preprocessing step ensures compatibility with the pipeline's statistical tests and interactive visualization.1213## When to use1415You have raw quantification data (abundance or intensity values across samples and features) from mass spectrometry or similar high-dimensional assays and need to prepare it for msFeaST's feature selection workflow, which requires standardized internal data structures compatible with R-based statistical tests (e.g., globaltest).1617## When NOT to use1819- Quantification data is already in msFeaST's internal format or has been validated and formatted by a prior preprocessing run.20- You only need to inspect pre-computed results without re-processing raw abundance data—use the dashboard directly with existing dashboard_data.json.21- Your quantification matrix lacks metadata or spectral data mappings, making alignment impossible.2223## Inputs2425- quantification table (CSV, TSV, or pandas DataFrame with samples × features abundance matrix)26- metadata table (for sample ID validation and alignment)27- data filepath arguments (user-specified locations of quantification files)2829## Outputs3031- normalized quantification data structure (Python dictionary or DataFrame in msFeaST internal format)32- validated sample-feature mappings ready for pipeline input33- intermediate data object compatible with R globaltest and statistical functions3435## How to apply3637Load the quantification table (typically a pandas DataFrame or CSV with samples as rows and features/metabolites as columns) using the msFeaST preprocessing notebook (preprocessing_mushroom_type_comparison.ipynb). Parse and validate the input data structure to ensure it meets msFeaST requirements—notably sample identifiers that map to metadata rows and feature identifiers that correspond to spectral data. Transform the quantification table into the internal Python data structure (dictionary or DataFrame) expected by msFeaST, which aligns with downstream R statistical dependencies including the globaltest package. The normalization step may include handling missing values, scaling, or log-transformation depending on the assay type; refer to the notebook's magenta-italicized user input sections for dataset-specific parameters. Validate alignment between quantification sample IDs and metadata table rows before proceeding to the pipeline.3839## Related tools4041- **msFeaST** (provides the preprocessing notebook, internal data format specification, and statistical pipeline that consumes the normalized quantification table) — https://github.com/kevinmildau/msFeaST42- **jupyter-notebook** (execution environment for the preprocessing_mushroom_type_comparison.ipynb notebook that loads, parses, and normalizes the quantification table)43- **pandas** (used implicitly within msFeaST preprocessing to load and manipulate quantification DataFrames)44- **globaltest (R package)** (downstream R dependency that receives the normalized quantification data for feature selection and statistical testing)4546## Examples4748```49jupyter-notebook preprocessing_mushroom_type_comparison.ipynb # then modify data filepath arguments to your quantification, metadata, and spectral data files and execute cells50```5152## Evaluation signals5354- Quantification table sample IDs match all rows in the metadata table with no missing or orphaned mappings.55- Feature identifiers in the quantification table align with feature IDs in the spectral data without duplicates or conflicts.56- The normalized data structure can be serialized and passed to the msFeaST pipeline without schema or type errors.57- R globaltest and associated statistical functions (dplyr, tibble) execute without error on the normalized quantification object.58- The pipeline produces a valid JSON output file that loads successfully in the msFeaST interactive dashboard.5960## Limitations6162- msFeaST preprocessing and pipeline workflow have been tested on macOS and Linux; Windows support is currently being worked on.63- R package installation via conda is not reliable for the required packages; remotes and BiocManager must be used to control R package versions, which may require several minutes for compilation.64- The skill assumes that sample identifiers and feature identifiers are stable and unambiguous across the quantification, metadata, and spectral data files; user must verify alignment before pipeline execution.65- No changelog is available to track changes in data format specifications between msFeaST versions, potentially causing compatibility issues if updating the tool.6667## Evidence6869- [other] Transform and normalize the quantification and metadata tables into the internal data format expected by msFeaST.: "Transform and normalize the quantification and metadata tables into the internal data format expected by msFeaST"70- [other] Parse and validate the input data structures to ensure compatibility with msFeaST requirements.: "Parse and validate the input data structures to ensure compatibility with msFeaST requirements"71- [readme] Navigate to the notebook folder and open the preprocessing_mushroom_type_comparison.ipynb and msfeast_pipeline_mushroom_type_comparison.ipynb notebooks on your local machine. These notebooks contain a complete example of quantification table, metadata table, and spectral data processing required for msFeaST, as well as a complete use-case example.: "These notebooks contain a complete example of quantification table, metadata table, and spectral data processing required for msFeaST"72- [readme] Text in magenta italics font highlights required user input for the pipeline.: "Text in magenta italics font highlights required user input for the pipeline"73- [readme] The current msFeaST pre-processing and pipeline workflow has been tested on macos and should work identically on linux operating systems. Windows support is currently being worked on.: "The current msFeaST pre-processing and pipeline workflow has been tested on macos and should work identically on linux operating systems. Windows support is currently being worked on"
