memory-efficient-pairwise-statistics
Summary
Compute Pearson correlation statistics across all pairs of mass spectrometry peaks without loading the entire pairwise matrix into memory. This skill splits large ion-pair datasets into sequential chunks, processes each in parallel, and returns correlation coefficients with Bonferroni-corrected significance flags.
When to use
You have an MSI intensity matrix (msimat object) and an annotated mass-difference table (massdiff object with known or hypothesized parent–adduct ion pairs) where the number of peak pairs exceeds available RAM. Apply this skill when you need to test whether candidate parent and adduct ions co-vary spatially across imaging pixels—a sign of true molecular adduction—but your dataset contains hundreds to thousands of peaks and direct pairwise correlation would exhaust memory.
When NOT to use
- Your dataset contains fewer than ~100 peak pairs and fits comfortably in available RAM; use corrPairsMSI directly instead.
- You have only a mass list (no pixel intensity values); correlation testing requires spatial co-abundance data across pixels.
- Your mass-difference table contains no annotated matches (no parent–adduct candidates); run adductMatch or topAdducts first to identify candidate pairs.
Inputs
- msimat object (MSI intensity matrix: pixels × masses)
- massdiff object (parent–adduct candidate pairs with matches column)
Outputs
- annotated massdiff object with appended columns: Estimate, P.value, Significance
How to apply
Load the MSI intensity matrix and the annotated massdiff object (containing a 'matches' column linking putative parent–adduct pairs). For datasets that fit in memory, call corrPairsMSI directly with the msimat and massdiff objects, specifying the default two-tailed Pearson correlation method. For larger datasets, invoke corrPairsMSIchunks instead, setting mem.limit (in GB) to a safe fraction of available RAM and ncores for parallel processing of each chunk. The function computes the Pearson correlation coefficient and raw p-value for each pair, then applies Bonferroni multiple-testing correction at α = 0.05 (default) to flag significant correlations. The output is the annotated massdiff object with three appended columns: Estimate (correlation coefficient), P.value (uncorrected), and Significance (boolean). Use this result to filter pairs showing statistically significant co-abundance patterns, which strengthens evidence for parent–adduct identity.
Related tools
- mass2adduct (R package providing corrPairsMSI and corrPairsMSIchunks functions for memory-efficient pairwise correlation testing on MSI data) — https://github.com/kbseah/mass2adduct
- R (Runtime environment for invoking corrPairsMSI or corrPairsMSIchunks; parallelization via ncores parameter)
- Cardinal (MSI data processing package; MSProcessedImagingExperiment and MSContinuousImagingExperiment objects can be converted to msimat format prior to correlation testing) — http://cardinalmsi.org/
Examples
d.diff.annot.cor <- corrPairsMSI(d, d.diff.annot); # Or for large datasets: d.diff.annot.cor <- corrPairsMSIchunks(d, d.diff.annot, mem.limit=4, ncores=4)
Evaluation signals
- The returned massdiff object has three new numeric columns (Estimate, P.value, Significance) with lengths equal to the number of input peak pairs.
- Correlation coefficients (Estimate) fall in the range [−1, 1]; p-values are non-negative and ≤ 1.0.
- Bonferroni-corrected significance flags (Significance = TRUE) are rare when the number of pairs is large, consistent with stringent multiple-testing correction.
- Pairs with high spatial co-abundance (Significance = TRUE) should show visually consistent spatial distributions when plotted side-by-side on the imaging tissue.
- Re-running the same analysis on the same input data with identical parameters (mem.limit, ncores) yields identical Estimate and P.value columns (chunking should not change numerical results, only memory efficiency).
Limitations
- Pearson correlation assumes linear relationships between parent and adduct ion intensities; nonlinear co-abundance patterns will not be detected.
- Bonferroni correction is conservative; with many pairs, the corrected significance threshold becomes very stringent, increasing false negatives.
- The msimunging.pl Perl script provided for converting large CSV files to triplet format is memory-limited by available RAM; datasets larger than available RAM may still fail even after conversion.
- Chunked processing (corrPairsMSIchunks) requires careful selection of mem.limit; if set too high, the process will crash; if set too low, speed gains from parallelization diminish due to excessive I/O overhead.
Evidence
- [other] corrPairsMSI performs a two-tailed correlation test using Pearson's method on each pair of peaks in the annotated mass-difference object, generating correlation statistics for parent and adduct ion intensity pairs.: "corrPairsMSI performs a two-tailed correlation test using Pearson's method on each pair of peaks in the annotated mass-difference object, generating correlation statistics for parent and adduct ion"
- [other] For datasets exceeding available memory, invoke corrPairsMSIchunks instead, specifying mem.limit in GB and ncores for parallel processing to chunk the pairwise correlations.: "For datasets exceeding available memory, invoke corrPairsMSIchunks instead, specifying mem.limit in GB and ncores for parallel processing to chunk the pairwise correlations."
- [other] Extract correlation results: Estimate (correlation coefficient), P.value (raw uncorrected p-value), and Significance (boolean flag after Bonferroni multiple-testing correction).: "Extract correlation results: Estimate (correlation coefficient), P.value (raw uncorrected p-value), and Significance (boolean flag after Bonferroni multiple-testing correction)."
- [readme] Output is a data.frame with p-values for each ion pair. By default the cutoff for significance is p=0.05 with Bonferroni correction.: "Output is a data.frame with p-values for each ion pair. By default the cutoff for significance is p=0.05 with Bonferroni correction."
- [readme] For large data sets, where the tables would not fit into memory, it is possible to break up the problem into "chunks" processed serially. Use the function
corrPairsMSIchunks instead of corrPairsMSI.: "For large data sets, where the tables would not fit into memory, it is possible to break up the problem into "chunks" processed serially. Use the function corrPairsMSIchunks instead of"
1---2name: memory-efficient-pairwise-statistics3description: Use when you have an MSI intensity matrix (msimat object) and an annotated mass-difference table (massdiff object with known or hypothesized parent–adduct ion pairs) where the number of peak pairs exceeds available RAM.4license: CC-BY-4.05---67# memory-efficient-pairwise-statistics89## Summary1011Compute Pearson correlation statistics across all pairs of mass spectrometry peaks without loading the entire pairwise matrix into memory. This skill splits large ion-pair datasets into sequential chunks, processes each in parallel, and returns correlation coefficients with Bonferroni-corrected significance flags.1213## When to use1415You have an MSI intensity matrix (msimat object) and an annotated mass-difference table (massdiff object with known or hypothesized parent–adduct ion pairs) where the number of peak pairs exceeds available RAM. Apply this skill when you need to test whether candidate parent and adduct ions co-vary spatially across imaging pixels—a sign of true molecular adduction—but your dataset contains hundreds to thousands of peaks and direct pairwise correlation would exhaust memory.1617## When NOT to use1819- Your dataset contains fewer than ~100 peak pairs and fits comfortably in available RAM; use corrPairsMSI directly instead.20- You have only a mass list (no pixel intensity values); correlation testing requires spatial co-abundance data across pixels.21- Your mass-difference table contains no annotated matches (no parent–adduct candidates); run adductMatch or topAdducts first to identify candidate pairs.2223## Inputs2425- msimat object (MSI intensity matrix: pixels × masses)26- massdiff object (parent–adduct candidate pairs with matches column)2728## Outputs2930- annotated massdiff object with appended columns: Estimate, P.value, Significance3132## How to apply3334Load the MSI intensity matrix and the annotated massdiff object (containing a 'matches' column linking putative parent–adduct pairs). For datasets that fit in memory, call corrPairsMSI directly with the msimat and massdiff objects, specifying the default two-tailed Pearson correlation method. For larger datasets, invoke corrPairsMSIchunks instead, setting mem.limit (in GB) to a safe fraction of available RAM and ncores for parallel processing of each chunk. The function computes the Pearson correlation coefficient and raw p-value for each pair, then applies Bonferroni multiple-testing correction at α = 0.05 (default) to flag significant correlations. The output is the annotated massdiff object with three appended columns: Estimate (correlation coefficient), P.value (uncorrected), and Significance (boolean). Use this result to filter pairs showing statistically significant co-abundance patterns, which strengthens evidence for parent–adduct identity.3536## Related tools3738- **mass2adduct** (R package providing corrPairsMSI and corrPairsMSIchunks functions for memory-efficient pairwise correlation testing on MSI data) — https://github.com/kbseah/mass2adduct39- **R** (Runtime environment for invoking corrPairsMSI or corrPairsMSIchunks; parallelization via ncores parameter)40- **Cardinal** (MSI data processing package; MSProcessedImagingExperiment and MSContinuousImagingExperiment objects can be converted to msimat format prior to correlation testing) — http://cardinalmsi.org/4142## Examples4344```45d.diff.annot.cor <- corrPairsMSI(d, d.diff.annot); # Or for large datasets: d.diff.annot.cor <- corrPairsMSIchunks(d, d.diff.annot, mem.limit=4, ncores=4)46```4748## Evaluation signals4950- The returned massdiff object has three new numeric columns (Estimate, P.value, Significance) with lengths equal to the number of input peak pairs.51- Correlation coefficients (Estimate) fall in the range [−1, 1]; p-values are non-negative and ≤ 1.0.52- Bonferroni-corrected significance flags (Significance = TRUE) are rare when the number of pairs is large, consistent with stringent multiple-testing correction.53- Pairs with high spatial co-abundance (Significance = TRUE) should show visually consistent spatial distributions when plotted side-by-side on the imaging tissue.54- Re-running the same analysis on the same input data with identical parameters (mem.limit, ncores) yields identical Estimate and P.value columns (chunking should not change numerical results, only memory efficiency).5556## Limitations5758- Pearson correlation assumes linear relationships between parent and adduct ion intensities; nonlinear co-abundance patterns will not be detected.59- Bonferroni correction is conservative; with many pairs, the corrected significance threshold becomes very stringent, increasing false negatives.60- The msimunging.pl Perl script provided for converting large CSV files to triplet format is memory-limited by available RAM; datasets larger than available RAM may still fail even after conversion.61- Chunked processing (corrPairsMSIchunks) requires careful selection of mem.limit; if set too high, the process will crash; if set too low, speed gains from parallelization diminish due to excessive I/O overhead.6263## Evidence6465- [other] corrPairsMSI performs a two-tailed correlation test using Pearson's method on each pair of peaks in the annotated mass-difference object, generating correlation statistics for parent and adduct ion intensity pairs.: "corrPairsMSI performs a two-tailed correlation test using Pearson's method on each pair of peaks in the annotated mass-difference object, generating correlation statistics for parent and adduct ion"66- [other] For datasets exceeding available memory, invoke corrPairsMSIchunks instead, specifying mem.limit in GB and ncores for parallel processing to chunk the pairwise correlations.: "For datasets exceeding available memory, invoke corrPairsMSIchunks instead, specifying mem.limit in GB and ncores for parallel processing to chunk the pairwise correlations."67- [other] Extract correlation results: Estimate (correlation coefficient), P.value (raw uncorrected p-value), and Significance (boolean flag after Bonferroni multiple-testing correction).: "Extract correlation results: Estimate (correlation coefficient), P.value (raw uncorrected p-value), and Significance (boolean flag after Bonferroni multiple-testing correction)."68- [readme] Output is a data.frame with p-values for each ion pair. By default the cutoff for significance is p=0.05 with Bonferroni correction.: "Output is a data.frame with p-values for each ion pair. By default the cutoff for significance is p=0.05 with Bonferroni correction."69- [readme] For large data sets, where the tables would not fit into memory, it is possible to break up the problem into "chunks" processed serially. Use the function `corrPairsMSIchunks` instead of `corrPairsMSI`.: "For large data sets, where the tables would not fit into memory, it is possible to break up the problem into "chunks" processed serially. Use the function `corrPairsMSIchunks` instead of"
HolobiomicsLab