matrix-structure-validation-and-qc
Summary
Validate the dimensionality, data types, feature/sample identifiers, and metadata attributes of MS2-derived sample matrices (e.g., MemoMatrix objects) to confirm correct construction and reproducibility. This skill ensures that fingerprint alignment and sample vectorization pipelines produce structurally sound outputs suitable for downstream analysis.
When to use
After executing memo_from_unaligned or memo_from_aligned functions to generate a MemoMatrix object from MS2 spectra or aligned feature tables. Use this skill to verify that the resulting matrix has the expected sample-by-feature structure, contains valid identifiers for all samples and MS2 peaks/neutral loss features, and retains necessary metadata attributes before proceeding to filtering, visualization, or comparative analysis.
When NOT to use
- Input is a raw MS2 spectra file (.mgf, .mzML, .mzXML) that has not yet been processed into a fingerprint matrix — use matchms and memo_from_unaligned/memo_from_aligned first.
- Input is an already-validated and production-ready MemoMatrix from a prior, verified run — re-validation is unnecessary unless pipeline parameters or upstream data changed.
- Goal is to filter out blank-associated peaks or perform statistical filtering — use filter_peaks_and_losses_from_blanks or visualization/dimensionality reduction techniques instead.
Inputs
- MemoMatrix object (output from memo_from_unaligned or memo_from_aligned)
- Reference MemoMatrix outputs from memo_publication_examples repository (optional, for comparison)
Outputs
- Validation report confirming correct dimensions, data types, and identifier presence
- Metadata attribute summary
- Comparison results against reference outputs (if applicable)
How to apply
Inspect the MemoMatrix artifact along multiple dimensions: (1) Verify dimensionality matches the expected number of samples and MS2 peak/neutral loss features counted across the dataset. (2) Check that all data types are numeric (counts) and consistent across the matrix. (3) Validate that sample identifiers and feature identifiers (MS2 peak m/z values or neutral loss mass differences) are present and non-duplicated. (4) Confirm that expected metadata attributes (e.g., precursor m/z, retention time information, sample source) are attached. (5) Compare the resulting matrix structure and content against reference outputs from the memo_publication_examples repository to confirm reproducibility. This validation is critical because MS2 fingerprints are generated by counting occurrences of MS2 peaks and neutral losses, so structural correctness directly impacts the validity of downstream sample comparisons.
Related tools
- memo-ms (Generates MemoMatrix objects from MS2 spectra or aligned feature tables via memo_from_unaligned and memo_from_aligned functions) — https://github.com/mandelbrot-project/memo
- matchms (Parses and processes MS2 spectra data from supported formats (.mgf, .mzML, .mzXML) prior to fingerprint generation) — https://github.com/matchms/matchms
- numpy (Provides matrix/array data structures and basic shape, type, and content inspection utilities)
- scikit-bio (Optional utility for advanced matrix structural and statistical validation)
- memo_publication_examples (Contains reference MemoMatrix outputs and notebooks for reproducibility comparison) — https://github.com/mandelbrot-project/memo_publication_examples
Examples
from memo import MemoMatrix; memo_obj = memo_from_unaligned(spectra_files); assert memo_obj.data.shape[0] == n_samples and memo_obj.data.shape[1] == n_unique_peaks; print(f'MemoMatrix shape: {memo_obj.data.shape}, data type: {memo_obj.data.dtype}')
Evaluation signals
- MemoMatrix shape (n_samples, n_features) matches the number of input samples and the total count of unique MS2 peaks and neutral losses detected.
- All matrix values are non-negative integers (counts), with no NaN, Inf, or unexpected data types.
- Sample identifiers (rows) are unique and map correctly to input sample files; feature identifiers (columns) are unique and correspond to MS2 m/z values or neutral loss mass differences.
- Metadata attributes (e.g., precursor m/z, source file, metadata fields from input) are present and populated for all samples.
- Comparison of matrix subset (e.g., first 5 rows, first 10 columns, row/column sums) against reference outputs from memo_publication_examples shows exact or near-identical values (allowing for minor version/environment differences).
Limitations
- Validation does not detect logical correctness of peak/loss counting if the upstream memo_from_unaligned or memo_from_aligned function contains bugs; it only checks structural properties.
- Reference outputs from memo_publication_examples may not be available for all datasets or parameter combinations, limiting full reproducibility comparison.
- Metadata attributes depend on completeness of input MS2 spectral metadata (e.g., precursor m/z, retention time); missing metadata in upstream files will result in incomplete MemoMatrix attributes.
- TMAP and interactive plot generation (mentioned in documentation) are only available on MacOS and Linux; Windows users require WSL.
Evidence
- [other] Inspect the resulting MemoMatrix for correct dimensionality, sample-by-feature structure, and presence of metadata attributes.: "Inspect the resulting MemoMatrix for correct dimensionality, sample-by-feature structure, and presence of metadata attributes."
- [other] Validate the resulting MemoMatrix artifact for correct dimensions, data types, and presence of expected feature and sample identifiers.: "Validate the resulting MemoMatrix artifact for correct dimensions, data types, and presence of expected feature and sample identifiers."
- [other] Compare matrix structure and content against reference outputs from the memo_publication_examples repository to confirm reproducibility.: "Compare matrix structure and content against reference outputs from the memo_publication_examples repository to confirm reproducibility."
- [intro] The occurence of MS2 peaks and neutral losses (to the precursor) in each sample is counted and used to generate an MS2 fingerprint of the sample: "The occurence of MS2 peaks and neutral losses (to the precursor) in each sample is counted and used to generate an MS2 fingerprint"
- [other] MEMO is mainly built on matchms and spec2vec packages for handling the MS2 spectra: "MEMO is mainly built on
matchms_ and spec2vec_ packages for handling the MS2 spectra"
1---2name: matrix-structure-validation-and-qc3description: Use when after executing memo_from_unaligned or memo_from_aligned functions to generate a MemoMatrix object from MS2 spectra or aligned feature tables.4license: CC-BY-4.05---67# matrix-structure-validation-and-qc89## Summary1011Validate the dimensionality, data types, feature/sample identifiers, and metadata attributes of MS2-derived sample matrices (e.g., MemoMatrix objects) to confirm correct construction and reproducibility. This skill ensures that fingerprint alignment and sample vectorization pipelines produce structurally sound outputs suitable for downstream analysis.1213## When to use1415After executing memo_from_unaligned or memo_from_aligned functions to generate a MemoMatrix object from MS2 spectra or aligned feature tables. Use this skill to verify that the resulting matrix has the expected sample-by-feature structure, contains valid identifiers for all samples and MS2 peaks/neutral loss features, and retains necessary metadata attributes before proceeding to filtering, visualization, or comparative analysis.1617## When NOT to use1819- Input is a raw MS2 spectra file (.mgf, .mzML, .mzXML) that has not yet been processed into a fingerprint matrix — use matchms and memo_from_unaligned/memo_from_aligned first.20- Input is an already-validated and production-ready MemoMatrix from a prior, verified run — re-validation is unnecessary unless pipeline parameters or upstream data changed.21- Goal is to filter out blank-associated peaks or perform statistical filtering — use filter_peaks_and_losses_from_blanks or visualization/dimensionality reduction techniques instead.2223## Inputs2425- MemoMatrix object (output from memo_from_unaligned or memo_from_aligned)26- Reference MemoMatrix outputs from memo_publication_examples repository (optional, for comparison)2728## Outputs2930- Validation report confirming correct dimensions, data types, and identifier presence31- Metadata attribute summary32- Comparison results against reference outputs (if applicable)3334## How to apply3536Inspect the MemoMatrix artifact along multiple dimensions: (1) Verify dimensionality matches the expected number of samples and MS2 peak/neutral loss features counted across the dataset. (2) Check that all data types are numeric (counts) and consistent across the matrix. (3) Validate that sample identifiers and feature identifiers (MS2 peak m/z values or neutral loss mass differences) are present and non-duplicated. (4) Confirm that expected metadata attributes (e.g., precursor m/z, retention time information, sample source) are attached. (5) Compare the resulting matrix structure and content against reference outputs from the memo_publication_examples repository to confirm reproducibility. This validation is critical because MS2 fingerprints are generated by counting occurrences of MS2 peaks and neutral losses, so structural correctness directly impacts the validity of downstream sample comparisons.3738## Related tools3940- **memo-ms** (Generates MemoMatrix objects from MS2 spectra or aligned feature tables via memo_from_unaligned and memo_from_aligned functions) — https://github.com/mandelbrot-project/memo41- **matchms** (Parses and processes MS2 spectra data from supported formats (.mgf, .mzML, .mzXML) prior to fingerprint generation) — https://github.com/matchms/matchms42- **numpy** (Provides matrix/array data structures and basic shape, type, and content inspection utilities)43- **scikit-bio** (Optional utility for advanced matrix structural and statistical validation)44- **memo_publication_examples** (Contains reference MemoMatrix outputs and notebooks for reproducibility comparison) — https://github.com/mandelbrot-project/memo_publication_examples4546## Examples4748```49from memo import MemoMatrix; memo_obj = memo_from_unaligned(spectra_files); assert memo_obj.data.shape[0] == n_samples and memo_obj.data.shape[1] == n_unique_peaks; print(f'MemoMatrix shape: {memo_obj.data.shape}, data type: {memo_obj.data.dtype}')50```5152## Evaluation signals5354- MemoMatrix shape (n_samples, n_features) matches the number of input samples and the total count of unique MS2 peaks and neutral losses detected.55- All matrix values are non-negative integers (counts), with no NaN, Inf, or unexpected data types.56- Sample identifiers (rows) are unique and map correctly to input sample files; feature identifiers (columns) are unique and correspond to MS2 m/z values or neutral loss mass differences.57- Metadata attributes (e.g., precursor m/z, source file, metadata fields from input) are present and populated for all samples.58- Comparison of matrix subset (e.g., first 5 rows, first 10 columns, row/column sums) against reference outputs from memo_publication_examples shows exact or near-identical values (allowing for minor version/environment differences).5960## Limitations6162- Validation does not detect logical correctness of peak/loss counting if the upstream memo_from_unaligned or memo_from_aligned function contains bugs; it only checks structural properties.63- Reference outputs from memo_publication_examples may not be available for all datasets or parameter combinations, limiting full reproducibility comparison.64- Metadata attributes depend on completeness of input MS2 spectral metadata (e.g., precursor m/z, retention time); missing metadata in upstream files will result in incomplete MemoMatrix attributes.65- TMAP and interactive plot generation (mentioned in documentation) are only available on MacOS and Linux; Windows users require WSL.6667## Evidence6869- [other] Inspect the resulting MemoMatrix for correct dimensionality, sample-by-feature structure, and presence of metadata attributes.: "Inspect the resulting MemoMatrix for correct dimensionality, sample-by-feature structure, and presence of metadata attributes."70- [other] Validate the resulting MemoMatrix artifact for correct dimensions, data types, and presence of expected feature and sample identifiers.: "Validate the resulting MemoMatrix artifact for correct dimensions, data types, and presence of expected feature and sample identifiers."71- [other] Compare matrix structure and content against reference outputs from the memo_publication_examples repository to confirm reproducibility.: "Compare matrix structure and content against reference outputs from the memo_publication_examples repository to confirm reproducibility."72- [intro] The occurence of MS2 peaks and neutral losses (to the precursor) in each sample is counted and used to generate an MS2 fingerprint of the sample: "The occurence of MS2 peaks and neutral losses (to the precursor) in each sample is counted and used to generate an MS2 fingerprint"73- [other] MEMO is mainly built on matchms and spec2vec packages for handling the MS2 spectra: "MEMO is mainly built on `matchms`_ and `spec2vec`_ packages for handling the MS2 spectra"
HolobiomicsLab 