statistical-significance-testing-p-value-calculation
Summary
Compute p-values and significance tests to quantify whether observed differences in link-scoring distributions (e.g., mean standardised scores for validated vs. random links) are unlikely under the null hypothesis of random association. This skill enables ranking and filtering of GCF-MF links by statistical strength.
When to use
After standardizing link scores (strain correlation and IOKR) across all potential GCF-MF pairs in a metabologenomics dataset, perform significance testing to determine whether validated links show statistically distinguishable score distributions from the background. Apply this when comparing mean or median scores between validated link subsets and all-link populations, or when filtering links by score percentile and need to confirm enrichment.
When NOT to use
- Input is already a pre-filtered or pre-ranked link set with no access to the full background distribution (p-values require population-level statistics).
- No validated ground-truth labels are available; significance testing requires a known subset of true links for comparison.
- Score distributions are highly non-normal and sample sizes are very small (<5 validated links); classical parametric tests may be unreliable.
Inputs
- standardised strain correlation scores for all GCF-MF link pairs
- standardised IOKR scores for all GCF-MF link pairs
- binary labels or ground-truth set of validated GCF-MF links
- link metadata (GCF identifier, MF identifier, score values)
- optional: score percentile threshold (e.g., 90th percentile)
Outputs
- p-values for pairwise score comparisons (validated vs. all links)
- mean and variance of scores stratified by link validation status
- significance test results with test statistic and effect size
- enrichment p-value for filtered link subsets
- ranked or prioritized link list annotated with significance labels
How to apply
For each scoring function (standardised strain correlation and IOKR), compute the mean score for the validated link subset and the full link population. Apply a statistical test (e.g., t-test, Mann–Whitney U, or permutation test) to generate a p-value quantifying the probability of observing the observed difference under random overlap. Report the p-value alongside effect size (mean difference). When filtering links above the 90th percentile by score, apply a hypergeometric or contingency-table test to assess whether the proportion of validated links in the filtered set exceeds the background rate, reporting the enrichment p-value. Use these p-values to prioritize scoring functions and combined scores that show the strongest statistical separation (lower p-values indicate stronger signal).
Related tools
- NPLinker (Framework that integrates genomic and metabolomic scoring functions and reports p-values for link enrichment validation) — https://github.com/sdrogers/nplinker
- BiG-SCAPE (Clusters BGCs into GCFs; output is necessary to compute strain correlation scores used in significance testing)
- antiSMASH (Detects BGCs from genomes; required upstream to define the GCF entities scored in link significance tests)
- GNPS (Provides MS/MS spectral data and metabolite identifications; used to train IOKR scoring model and define validated MF-spectrum pairs)
Evaluation signals
- p-values for validated link subset are much smaller (e.g., p < 0.001) than for background all-link population, indicating strong statistical separation of true from random links.
- Mean standardised score for validated links is >2 standard deviations from the mean of all links (e.g., validated mean 3.67, all-link mean −0.006 as reported).
- When links are filtered above the 90th percentile, the proportion of validated links in the filtered subset significantly exceeds the background rate (enrichment p-value < 0.05).
- Combined or multi-score filtering (both IOKR and standardised correlation >90th percentile) yields lower p-values than either score alone, confirming complementarity.
- Statistical test assumptions are met: if using parametric tests, verify normality of score distributions; if non-normal, confirm use of non-parametric alternatives (e.g., Mann–Whitney U) or report effect sizes alongside p-values.
Limitations
- Standardised strain correlation score still suffers from inability to distinguish between links showing identical strain presence/absence patterns, limiting the biological interpretability of marginal p-value differences.
- IOKR scoring relies on MIBiG homology; validated link sets used for significance testing are biased toward BGCs with known close homologs, potentially inflating p-values for novel or divergent BGCs.
- P-value computation assumes independence of link pairs; in reality, GCFs and MFs are nested structures, so multiple links per GCF or MF may violate independence assumptions and inflate significance.
- Sample size of validated links is often small (e.g., 8 validated links in Crüsemann dataset); statistical power is limited, and p-values may be unstable or driven by outliers.
Evidence
- [other] The standardised strain correlation score has a mean of -0.0060 and variance of 1 across all links (compared to raw score mean of 83.5144), with standardised scores for validated links reaching 3.6717 (p=6.8302 × 10−64).: "The standardised strain correlation score has a mean of -0.0060 and variance of 1 across all links (compared to raw score mean of 83.5144), with standardised scores for validated links reaching"
- [results] For the raw strain correlation score, the mean score is 83.514 for all links, and 14.667 for validated links: "For the raw strain correlation score, the mean score is 83.514 for all links, and 14.667 for validated links"
- [results] Standardising the score gives a mean score of -0.006 for all links, and 3.672 for validated links: "Standardising the score gives a mean score of -0.006 for all links, and 3.672 for validated links"
- [results] the set of links scoring above the 90th percentile on both scores is significantly enriched compared to the set that exceed either of the individual scores (p-value of 2.633 × 10−4 and 0.0208: "the set of links scoring above the 90th percentile on both scores is significantly enriched compared to the set that exceed either of the individual scores (p-value of 2.633 × 10−4 and 0.0208"
- [results] both the IOKR and standardised strain correlation scores are significantly enriched (p-value of 0.0139 and 2.483 × 10−11, respectively): "both the IOKR and standardised strain correlation scores are significantly enriched (p-value of 0.0139 and 2.483 × 10−11, respectively)"
- [other] For each link, calculate the hypergeometric expected value E[σ_corr(M,G)] and variance Var[σ_corr(M,G)] under the null hypothesis of random overlap, using population size N, MF size m, GCF size g, and overlap count o.: "For each link, calculate the hypergeometric expected value E[σ_corr(M,G)] and variance Var[σ_corr(M,G)] under the null hypothesis of random overlap"
1---2name: statistical-significance-testing-p-value-calculation3description: Use when after standardizing link scores (strain correlation and IOKR) across all potential GCF-MF pairs in a metabologenomics dataset, perform significance testing to determine whether validated links show statistically distinguishable score distributions from the background.4license: CC-BY-4.05---67# statistical-significance-testing-p-value-calculation89## Summary1011Compute p-values and significance tests to quantify whether observed differences in link-scoring distributions (e.g., mean standardised scores for validated vs. random links) are unlikely under the null hypothesis of random association. This skill enables ranking and filtering of GCF-MF links by statistical strength.1213## When to use1415After standardizing link scores (strain correlation and IOKR) across all potential GCF-MF pairs in a metabologenomics dataset, perform significance testing to determine whether validated links show statistically distinguishable score distributions from the background. Apply this when comparing mean or median scores between validated link subsets and all-link populations, or when filtering links by score percentile and need to confirm enrichment.1617## When NOT to use1819- Input is already a pre-filtered or pre-ranked link set with no access to the full background distribution (p-values require population-level statistics).20- No validated ground-truth labels are available; significance testing requires a known subset of true links for comparison.21- Score distributions are highly non-normal and sample sizes are very small (<5 validated links); classical parametric tests may be unreliable.2223## Inputs2425- standardised strain correlation scores for all GCF-MF link pairs26- standardised IOKR scores for all GCF-MF link pairs27- binary labels or ground-truth set of validated GCF-MF links28- link metadata (GCF identifier, MF identifier, score values)29- optional: score percentile threshold (e.g., 90th percentile)3031## Outputs3233- p-values for pairwise score comparisons (validated vs. all links)34- mean and variance of scores stratified by link validation status35- significance test results with test statistic and effect size36- enrichment p-value for filtered link subsets37- ranked or prioritized link list annotated with significance labels3839## How to apply4041For each scoring function (standardised strain correlation and IOKR), compute the mean score for the validated link subset and the full link population. Apply a statistical test (e.g., t-test, Mann–Whitney U, or permutation test) to generate a p-value quantifying the probability of observing the observed difference under random overlap. Report the p-value alongside effect size (mean difference). When filtering links above the 90th percentile by score, apply a hypergeometric or contingency-table test to assess whether the proportion of validated links in the filtered set exceeds the background rate, reporting the enrichment p-value. Use these p-values to prioritize scoring functions and combined scores that show the strongest statistical separation (lower p-values indicate stronger signal).4243## Related tools4445- **NPLinker** (Framework that integrates genomic and metabolomic scoring functions and reports p-values for link enrichment validation) — https://github.com/sdrogers/nplinker46- **BiG-SCAPE** (Clusters BGCs into GCFs; output is necessary to compute strain correlation scores used in significance testing)47- **antiSMASH** (Detects BGCs from genomes; required upstream to define the GCF entities scored in link significance tests)48- **GNPS** (Provides MS/MS spectral data and metabolite identifications; used to train IOKR scoring model and define validated MF-spectrum pairs)4950## Evaluation signals5152- p-values for validated link subset are much smaller (e.g., p < 0.001) than for background all-link population, indicating strong statistical separation of true from random links.53- Mean standardised score for validated links is >2 standard deviations from the mean of all links (e.g., validated mean 3.67, all-link mean −0.006 as reported).54- When links are filtered above the 90th percentile, the proportion of validated links in the filtered subset significantly exceeds the background rate (enrichment p-value < 0.05).55- Combined or multi-score filtering (both IOKR and standardised correlation >90th percentile) yields lower p-values than either score alone, confirming complementarity.56- Statistical test assumptions are met: if using parametric tests, verify normality of score distributions; if non-normal, confirm use of non-parametric alternatives (e.g., Mann–Whitney U) or report effect sizes alongside p-values.5758## Limitations5960- Standardised strain correlation score still suffers from inability to distinguish between links showing identical strain presence/absence patterns, limiting the biological interpretability of marginal p-value differences.61- IOKR scoring relies on MIBiG homology; validated link sets used for significance testing are biased toward BGCs with known close homologs, potentially inflating p-values for novel or divergent BGCs.62- P-value computation assumes independence of link pairs; in reality, GCFs and MFs are nested structures, so multiple links per GCF or MF may violate independence assumptions and inflate significance.63- Sample size of validated links is often small (e.g., 8 validated links in Crüsemann dataset); statistical power is limited, and p-values may be unstable or driven by outliers.6465## Evidence6667- [other] The standardised strain correlation score has a mean of -0.0060 and variance of 1 across all links (compared to raw score mean of 83.5144), with standardised scores for validated links reaching 3.6717 (p=6.8302 × 10−64).: "The standardised strain correlation score has a mean of -0.0060 and variance of 1 across all links (compared to raw score mean of 83.5144), with standardised scores for validated links reaching"68- [results] For the raw strain correlation score, the mean score is 83.514 for all links, and 14.667 for validated links: "For the raw strain correlation score, the mean score is 83.514 for all links, and 14.667 for validated links"69- [results] Standardising the score gives a mean score of -0.006 for all links, and 3.672 for validated links: "Standardising the score gives a mean score of -0.006 for all links, and 3.672 for validated links"70- [results] the set of links scoring above the 90th percentile on both scores is significantly enriched compared to the set that exceed either of the individual scores (p-value of 2.633 × 10−4 and 0.0208: "the set of links scoring above the 90th percentile on both scores is significantly enriched compared to the set that exceed either of the individual scores (p-value of 2.633 × 10−4 and 0.0208"71- [results] both the IOKR and standardised strain correlation scores are significantly enriched (p-value of 0.0139 and 2.483 × 10−11, respectively): "both the IOKR and standardised strain correlation scores are significantly enriched (p-value of 0.0139 and 2.483 × 10−11, respectively)"72- [other] For each link, calculate the hypergeometric expected value E[σ_corr(M,G)] and variance Var[σ_corr(M,G)] under the null hypothesis of random overlap, using population size N, MF size m, GCF size g, and overlap count o.: "For each link, calculate the hypergeometric expected value E[σ_corr(M,G)] and variance Var[σ_corr(M,G)] under the null hypothesis of random overlap"
HolobiomicsLab