Validated-Link Proportional Comparison
Summary
A statistical method to compare the enrichment of validated genomic-metabolomic links across partitioned scoring categories, using proportional analysis and Fisher exact tests to assess whether links meeting combined scoring thresholds are significantly more reliable than links meeting individual thresholds alone.
When to use
When you have scored GCF-MF (gene cluster family–molecular family) links using two or more complementary scoring approaches (e.g., standardised strain correlation and IOKR), and you need to determine whether combining scores improves discrimination of true links from false positives. Specifically, use this skill when you have ground-truth validated links and want to test whether links exceeding multiple score thresholds simultaneously show significantly higher validation rates than links exceeding either threshold alone.
When NOT to use
- You have only one scoring function available; the method requires at least two independent scoring approaches to assess complementarity.
- You lack ground-truth validated links; Fisher exact tests require a binary classification (validated/not validated) for each link.
- Your link scores are not independently derived; if the two scoring functions are highly correlated or derived from the same underlying data, the assumption of complementarity is violated.
Inputs
- GCF-MF link scores (two independent scoring functions, e.g., standardised strain correlation and IOKR scores)
- List of validated links (ground-truth positive pairs)
- Link partition thresholds (percentile values, typically 90th percentile)
Outputs
- Proportion of validated links per category (single score A, single score B, both scores, neither)
- Fisher exact test p-values comparing dual-threshold to each single-threshold category
- Pooled summary table with per-dataset and cross-dataset results
How to apply
Partition all GCF-MF links into four mutually exclusive categories based on independent percentile thresholds (e.g., 90th percentile) for each scoring function: (1) above threshold for score A only, (2) above threshold for score B only, (3) above both thresholds, (4) below both thresholds. For each category, calculate the proportion of validated links (validated count ÷ total links in category). Pool data across independent datasets (e.g., Crüsemann, Gross, Leão) and compute two-tailed Fisher exact test p-values comparing the dual-threshold category proportion to each single-threshold category proportion. Tabulate results showing proportions and p-values for each dataset individually and for the pooled analysis. The rationale is that complementarity between scoring functions is demonstrated when the dual-threshold category exhibits significantly enriched validation rates compared to single-threshold sets, indicating that the two approaches capture different aspects of link quality.
Related tools
- NPLinker (Framework integrating genomics and metabolomics data; orchestrates link scoring and provides GCF-MF link inputs for proportional comparison) — https://github.com/sdrogers/nplinker
- antiSMASH (Detects biosynthetic gene clusters (BGCs) from microbial genomes; outputs are clustered into GCFs used to define genomic entities in link scoring)
- BiG-SCAPE (Clusters BGCs into gene cluster families (GCFs); GCF membership is required input for link scoring and partitioning)
- MIBiG (Reference database of characterized BGCs; used by IOKR scoring and provides ground-truth validated BGC–metabolite pairs for analysis)
- GNPS (Public metabolomics spectral database; provides MS2 spectra as training data for IOKR and spectra used in link scoring)
Evaluation signals
- Verify that the four partition categories are mutually exclusive and collectively exhaustive (all links assigned to exactly one category).
- Confirm that the dual-threshold category proportion of validated links is statistically significantly higher than both single-threshold category proportions (p < 0.05), demonstrating complementarity.
- Check that within-dataset p-values and pooled p-values show consistent direction of effect (dual-threshold consistently enriched) across all datasets.
- Validate that proportions are calculated correctly: (count of validated links in category) ÷ (total links in category), not a ratio of validated to unvalidated.
- Ensure Fisher exact test assumptions are met: categories have sufficient sample sizes (typically n ≥ 5 per cell in contingency table) or report exact p-values for sparse counts.
Limitations
- The method depends critically on the quality and completeness of ground-truth validated links; sparse or biased validation sets can distort enrichment estimates.
- Independence of the two scoring functions is assumed; if scores are correlated or share underlying features, complementarity may be overstated and p-values inflated.
- Percentile thresholds are arbitrary (e.g., 90th percentile); results are sensitive to threshold choice, and no principled method for selecting optimal thresholds is provided in the article.
- Standardised strain correlation scores still suffer from inability to distinguish links with identical patterns of strain presence/absence, limiting discrimination even after standardisation.
- IOKR scoring requires BGC–MIBiG homology to assign molecular structures, restricting its applicability to BGCs with high sequence similarity to reference clusters.
Evidence
- [other] Partition links into four categories: above 90th percentile for standardised correlation only, above 90th percentile for IOKR only, above 90th percentile for both scores, and below 90th percentile for both.: "Partition links into four categories: above 90th percentile for standardised correlation only, above 90th percentile for IOKR only, above 90th percentile for both scores, and below 90th percentile"
- [other] For each category, compute the proportion of validated links (number of validated links in category / total links in category).: "For each category, compute the proportion of validated links (number of validated links in category / total links in category)."
- [other] Pool data across all three datasets and compute Fisher exact test p-values comparing the proportion of validated links in the dual-score (both scores ≥90th percentile) category to each single-score category.: "Pool data across all three datasets and compute Fisher exact test p-values comparing the proportion of validated links in the dual-score (both scores ≥90th percentile) category to each single-score"
- [results] Links scoring above the 90th percentile on both standardised strain correlation and IOKR scores are significantly enriched for validated links (p-value 2.633 × 10−4 from IOKR and 0.0208 from standardised strain correlation), demonstrating complementarity of the two scoring approaches.: "Links scoring above the 90th percentile on both standardised strain correlation and IOKR scores are significantly enriched for validated links (p-value 2.633 × 10−4 from IOKR and 0.0208 from"
- [abstract] using multiple link-scoring functions together makes it easier to prioritise true links relative to others: "using multiple link-scoring functions together makes it easier to prioritise true links relative to others"
- [other] Tabulate results showing proportions and p-values for all three datasets and pooled analysis, matching the structure and values reported in Table 2 and Table D in S1 Text.: "Tabulate results showing proportions and p-values for all three datasets and pooled analysis, matching the structure and values reported in Table 2 and Table D in S1 Text."
1---2name: validated-link-proportional-comparison3description: Use when when you have scored GCF-MF (gene cluster family–molecular family) links using two or more complementary scoring approaches (e.g., standardised strain correlation and IOKR), and you need to determine whether combining scores improves discrimination of true links from false positives.4license: CC-BY-4.05---67# Validated-Link Proportional Comparison89## Summary1011A statistical method to compare the enrichment of validated genomic-metabolomic links across partitioned scoring categories, using proportional analysis and Fisher exact tests to assess whether links meeting combined scoring thresholds are significantly more reliable than links meeting individual thresholds alone.1213## When to use1415When you have scored GCF-MF (gene cluster family–molecular family) links using two or more complementary scoring approaches (e.g., standardised strain correlation and IOKR), and you need to determine whether combining scores improves discrimination of true links from false positives. Specifically, use this skill when you have ground-truth validated links and want to test whether links exceeding multiple score thresholds simultaneously show significantly higher validation rates than links exceeding either threshold alone.1617## When NOT to use1819- You have only one scoring function available; the method requires at least two independent scoring approaches to assess complementarity.20- You lack ground-truth validated links; Fisher exact tests require a binary classification (validated/not validated) for each link.21- Your link scores are not independently derived; if the two scoring functions are highly correlated or derived from the same underlying data, the assumption of complementarity is violated.2223## Inputs2425- GCF-MF link scores (two independent scoring functions, e.g., standardised strain correlation and IOKR scores)26- List of validated links (ground-truth positive pairs)27- Link partition thresholds (percentile values, typically 90th percentile)2829## Outputs3031- Proportion of validated links per category (single score A, single score B, both scores, neither)32- Fisher exact test p-values comparing dual-threshold to each single-threshold category33- Pooled summary table with per-dataset and cross-dataset results3435## How to apply3637Partition all GCF-MF links into four mutually exclusive categories based on independent percentile thresholds (e.g., 90th percentile) for each scoring function: (1) above threshold for score A only, (2) above threshold for score B only, (3) above both thresholds, (4) below both thresholds. For each category, calculate the proportion of validated links (validated count ÷ total links in category). Pool data across independent datasets (e.g., Crüsemann, Gross, Leão) and compute two-tailed Fisher exact test p-values comparing the dual-threshold category proportion to each single-threshold category proportion. Tabulate results showing proportions and p-values for each dataset individually and for the pooled analysis. The rationale is that complementarity between scoring functions is demonstrated when the dual-threshold category exhibits significantly enriched validation rates compared to single-threshold sets, indicating that the two approaches capture different aspects of link quality.3839## Related tools4041- **NPLinker** (Framework integrating genomics and metabolomics data; orchestrates link scoring and provides GCF-MF link inputs for proportional comparison) — https://github.com/sdrogers/nplinker42- **antiSMASH** (Detects biosynthetic gene clusters (BGCs) from microbial genomes; outputs are clustered into GCFs used to define genomic entities in link scoring)43- **BiG-SCAPE** (Clusters BGCs into gene cluster families (GCFs); GCF membership is required input for link scoring and partitioning)44- **MIBiG** (Reference database of characterized BGCs; used by IOKR scoring and provides ground-truth validated BGC–metabolite pairs for analysis)45- **GNPS** (Public metabolomics spectral database; provides MS2 spectra as training data for IOKR and spectra used in link scoring)4647## Evaluation signals4849- Verify that the four partition categories are mutually exclusive and collectively exhaustive (all links assigned to exactly one category).50- Confirm that the dual-threshold category proportion of validated links is statistically significantly higher than both single-threshold category proportions (p < 0.05), demonstrating complementarity.51- Check that within-dataset p-values and pooled p-values show consistent direction of effect (dual-threshold consistently enriched) across all datasets.52- Validate that proportions are calculated correctly: (count of validated links in category) ÷ (total links in category), not a ratio of validated to unvalidated.53- Ensure Fisher exact test assumptions are met: categories have sufficient sample sizes (typically n ≥ 5 per cell in contingency table) or report exact p-values for sparse counts.5455## Limitations5657- The method depends critically on the quality and completeness of ground-truth validated links; sparse or biased validation sets can distort enrichment estimates.58- Independence of the two scoring functions is assumed; if scores are correlated or share underlying features, complementarity may be overstated and p-values inflated.59- Percentile thresholds are arbitrary (e.g., 90th percentile); results are sensitive to threshold choice, and no principled method for selecting optimal thresholds is provided in the article.60- Standardised strain correlation scores still suffer from inability to distinguish links with identical patterns of strain presence/absence, limiting discrimination even after standardisation.61- IOKR scoring requires BGC–MIBiG homology to assign molecular structures, restricting its applicability to BGCs with high sequence similarity to reference clusters.6263## Evidence6465- [other] Partition links into four categories: above 90th percentile for standardised correlation only, above 90th percentile for IOKR only, above 90th percentile for both scores, and below 90th percentile for both.: "Partition links into four categories: above 90th percentile for standardised correlation only, above 90th percentile for IOKR only, above 90th percentile for both scores, and below 90th percentile"66- [other] For each category, compute the proportion of validated links (number of validated links in category / total links in category).: "For each category, compute the proportion of validated links (number of validated links in category / total links in category)."67- [other] Pool data across all three datasets and compute Fisher exact test p-values comparing the proportion of validated links in the dual-score (both scores ≥90th percentile) category to each single-score category.: "Pool data across all three datasets and compute Fisher exact test p-values comparing the proportion of validated links in the dual-score (both scores ≥90th percentile) category to each single-score"68- [results] Links scoring above the 90th percentile on both standardised strain correlation and IOKR scores are significantly enriched for validated links (p-value 2.633 × 10−4 from IOKR and 0.0208 from standardised strain correlation), demonstrating complementarity of the two scoring approaches.: "Links scoring above the 90th percentile on both standardised strain correlation and IOKR scores are significantly enriched for validated links (p-value 2.633 × 10−4 from IOKR and 0.0208 from"69- [abstract] using multiple link-scoring functions together makes it easier to prioritise true links relative to others: "using multiple link-scoring functions together makes it easier to prioritise true links relative to others"70- [other] Tabulate results showing proportions and p-values for all three datasets and pooled analysis, matching the structure and values reported in Table 2 and Table D in S1 Text.: "Tabulate results showing proportions and p-values for all three datasets and pooled analysis, matching the structure and values reported in Table 2 and Table D in S1 Text."
HolobiomicsLab