spectral-scoring-and-significance-assessment
Summary
Evaluate and compare database search results from natural product mass spectrometry pipelines by computing match scores, p-values, and false discovery rates (FDR) to distinguish true identifications from noise. This skill is essential for validating which database search pipeline (Dereplicator, VarQuest, Dereplicator+) provides reliable identifications and assessing the statistical confidence of spectrum-to-structure matches.
When to use
You have executed multiple database search pipelines (Dereplicator, VarQuest, and/or Dereplicator+) on centroided LC-MS/MS spectra (in MGF, mzXML, mzML, or mzData format) against a chemical structure database of known natural products, and you need to: (1) identify which pipeline(s) successfully detected matches in your test spectra, (2) rank matches by statistical significance rather than raw score alone, (3) cross-compare hit patterns and sensitivity across search modes (e.g., standard vs. blind/unbiased modification search), or (4) control false discovery rate when reporting identifications.
When NOT to use
- Spectra have not yet been searched against a database (this skill assumes search results already exist; it is a post-processing step, not a search tool itself).
- You are using only a single database search pipeline and have no cross-pipeline comparison goal — in that case, focus on single-tool result interpretation rather than comparative analysis.
- Raw, uncentroided spectra or spectra in proprietary vendor formats that have not been converted to MGF/mzXML/mzML — preprocessing and format conversion must precede this skill.
Inputs
- significant_matches.tsv output from Dereplicator pipeline
- significant_matches.tsv output from VarQuest pipeline
- significant_matches.tsv output from Dereplicator+ pipeline
- Scan identifiers from test spectra (MSV000080102 .mzML files converted to MGF)
- Match scores (numeric)
- P-values (numeric, one per match)
- False discovery rates (numeric, one per match)
Outputs
- Cross-compared hit set table (scan ID, pipeline, score, p-value, FDR, common vs. unique)
- Tool-specific detections list (matches found by only one pipeline)
- Common identifications list (matches found by ≥2 pipelines)
- Mode-dependent sensitivity analysis (standard vs. blind/modification-tolerant mode comparison)
- Ranked match list filtered by p-value and FDR thresholds
How to apply
Parse the significant_matches.tsv output files generated by each database search tool, extracting scan identifiers, match scores, p-values, and false discovery rates for each spectrum–structure pair. Tabulate these metrics side-by-side to enable tool-to-tool comparison. Filter matches using both score and statistical thresholds (p-value and FDR cutoffs should be justified based on your false positive tolerance). Identify tool-specific detections (matches unique to one pipeline), common identifications (hits shared across pipelines, which increase confidence), and mode-dependent sensitivity differences (e.g., standard Dereplicator vs. blind-mode MetaMiner). Cross-reference scan identifiers to ensure you are comparing the same spectra across tools. Use p-values and FDR to rank and report matches in order of statistical confidence rather than raw score alone, as this guards against spurious high-scoring alignments.
Related tools
- Dereplicator (database search pipeline for peptidic natural products; generates match scores and results for tabulation and cross-comparison) — https://github.com/ablab/npdtools
- VarQuest (modification-tolerant database search pipeline; produces alternative hits and sensitivity profile for comparison with Dereplicator and Dereplicator+) — https://github.com/ablab/npdtools
- Dereplicator+ (extended database search pipeline for both peptidic and non-peptidic metabolites; provides an alternative hit set for cross-validation) — https://github.com/ablab/npdtools
- MetaMiner (metabologenomic pipeline that includes database search via Dereplicator; supports standard and blind (unbiased modification) search modes for mode-dependent sensitivity comparison) — https://github.com/ablab/npdtools
- Python (scripting language used to parse significant_matches.tsv, tabulate results, and cross-compare metrics across tools)
Examples
python parse_matches.py --dereplicator significant_matches.tsv --varquest significant_matches.tsv --dereplicate_plus significant_matches.tsv --output cross_comparison.tsv
Evaluation signals
- Cross-comparison table is complete: every scan ID from the test spectra appears in the output with entries for all pipelines that detected a match (or an explicit 'no match' entry).
- P-values and FDR values are present and finite (not NaN or infinite) for all reported matches; FDR is monotonically non-decreasing when matches are sorted by p-value within each tool.
- Tool-specific and common identification lists are mutually exclusive and exhaustive (every match belongs to exactly one category: unique to one tool, or shared by ≥2 tools).
- Matches ranked by p-value show expected rank order (lower p-value = more confident match); matches with identical p-values can be sub-ranked by score if reported.
- Mode-dependent comparison (standard vs. blind search) shows that blind/modification-tolerant modes detect additional matches not found in standard mode, validating the added sensitivity of flexible search.
Limitations
- P-values and FDR are meaningful only if the database search pipeline computed them; if a pipeline outputs only raw scores without statistical tests, comparison requires separate statistical modeling (e.g., permutation or null-model fitting).
- Cross-pipeline comparison assumes all tools searched the same spectrum file set and the same (or comparable) chemical structure database; mismatches in database version or content will confound interpretation of tool-specific vs. true differences.
- Scan identifier matching across tool outputs requires careful attention to format and naming conventions; if pipelines report scan IDs differently (e.g., with or without file prefixes), a mapping step is necessary before tabulation.
- False discovery rate control is valid only if applied uniformly across all comparisons; if tools use different FDR estimation methods or null models, direct FDR comparison may be misleading.
- Mode-dependent sensitivity assessment (e.g., standard vs. blind MetaMiner) is only informative if both modes are run on identical input spectra and with matched search parameters; parameter differences can dominate mode effects.
Evidence
- [methods] Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results: "Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results."
- [methods] Execute VarQuest pipeline on the same inputs with standard parameters: "Execute VarQuest pipeline on the same inputs with standard parameters."
- [methods] Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates: "Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates."
- [methods] Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences: "Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences."
- [readme] Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra: "Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra"
- [readme] VarQuest — a tool for modification-tolerant identification of novel variants of PNPs: "VarQuest — a tool for modification-tolerant identification of novel variants of PNPs"
- [readme] Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra: "Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra"
1---2name: spectral-scoring-and-significance-assessment3description: Use when you have executed multiple database search pipelines (Dereplicator, VarQuest, and/or Dereplicator+) on centroided LC-MS/MS spectra (in MGF, mzXML, mzML, or mzData format) against a chemical structure database of known natural products, and you need to: (1) identify which pipeline(s).4license: CC-BY-4.05---67# spectral-scoring-and-significance-assessment89## Summary1011Evaluate and compare database search results from natural product mass spectrometry pipelines by computing match scores, p-values, and false discovery rates (FDR) to distinguish true identifications from noise. This skill is essential for validating which database search pipeline (Dereplicator, VarQuest, Dereplicator+) provides reliable identifications and assessing the statistical confidence of spectrum-to-structure matches.1213## When to use1415You have executed multiple database search pipelines (Dereplicator, VarQuest, and/or Dereplicator+) on centroided LC-MS/MS spectra (in MGF, mzXML, mzML, or mzData format) against a chemical structure database of known natural products, and you need to: (1) identify which pipeline(s) successfully detected matches in your test spectra, (2) rank matches by statistical significance rather than raw score alone, (3) cross-compare hit patterns and sensitivity across search modes (e.g., standard vs. blind/unbiased modification search), or (4) control false discovery rate when reporting identifications.1617## When NOT to use1819- Spectra have not yet been searched against a database (this skill assumes search results already exist; it is a post-processing step, not a search tool itself).20- You are using only a single database search pipeline and have no cross-pipeline comparison goal — in that case, focus on single-tool result interpretation rather than comparative analysis.21- Raw, uncentroided spectra or spectra in proprietary vendor formats that have not been converted to MGF/mzXML/mzML — preprocessing and format conversion must precede this skill.2223## Inputs2425- significant_matches.tsv output from Dereplicator pipeline26- significant_matches.tsv output from VarQuest pipeline27- significant_matches.tsv output from Dereplicator+ pipeline28- Scan identifiers from test spectra (MSV000080102 .mzML files converted to MGF)29- Match scores (numeric)30- P-values (numeric, one per match)31- False discovery rates (numeric, one per match)3233## Outputs3435- Cross-compared hit set table (scan ID, pipeline, score, p-value, FDR, common vs. unique)36- Tool-specific detections list (matches found by only one pipeline)37- Common identifications list (matches found by ≥2 pipelines)38- Mode-dependent sensitivity analysis (standard vs. blind/modification-tolerant mode comparison)39- Ranked match list filtered by p-value and FDR thresholds4041## How to apply4243Parse the significant_matches.tsv output files generated by each database search tool, extracting scan identifiers, match scores, p-values, and false discovery rates for each spectrum–structure pair. Tabulate these metrics side-by-side to enable tool-to-tool comparison. Filter matches using both score and statistical thresholds (p-value and FDR cutoffs should be justified based on your false positive tolerance). Identify tool-specific detections (matches unique to one pipeline), common identifications (hits shared across pipelines, which increase confidence), and mode-dependent sensitivity differences (e.g., standard Dereplicator vs. blind-mode MetaMiner). Cross-reference scan identifiers to ensure you are comparing the same spectra across tools. Use p-values and FDR to rank and report matches in order of statistical confidence rather than raw score alone, as this guards against spurious high-scoring alignments.4445## Related tools4647- **Dereplicator** (database search pipeline for peptidic natural products; generates match scores and results for tabulation and cross-comparison) — https://github.com/ablab/npdtools48- **VarQuest** (modification-tolerant database search pipeline; produces alternative hits and sensitivity profile for comparison with Dereplicator and Dereplicator+) — https://github.com/ablab/npdtools49- **Dereplicator+** (extended database search pipeline for both peptidic and non-peptidic metabolites; provides an alternative hit set for cross-validation) — https://github.com/ablab/npdtools50- **MetaMiner** (metabologenomic pipeline that includes database search via Dereplicator; supports standard and blind (unbiased modification) search modes for mode-dependent sensitivity comparison) — https://github.com/ablab/npdtools51- **Python** (scripting language used to parse significant_matches.tsv, tabulate results, and cross-compare metrics across tools)5253## Examples5455```56python parse_matches.py --dereplicator significant_matches.tsv --varquest significant_matches.tsv --dereplicate_plus significant_matches.tsv --output cross_comparison.tsv57```5859## Evaluation signals6061- Cross-comparison table is complete: every scan ID from the test spectra appears in the output with entries for all pipelines that detected a match (or an explicit 'no match' entry).62- P-values and FDR values are present and finite (not NaN or infinite) for all reported matches; FDR is monotonically non-decreasing when matches are sorted by p-value within each tool.63- Tool-specific and common identification lists are mutually exclusive and exhaustive (every match belongs to exactly one category: unique to one tool, or shared by ≥2 tools).64- Matches ranked by p-value show expected rank order (lower p-value = more confident match); matches with identical p-values can be sub-ranked by score if reported.65- Mode-dependent comparison (standard vs. blind search) shows that blind/modification-tolerant modes detect additional matches not found in standard mode, validating the added sensitivity of flexible search.6667## Limitations6869- P-values and FDR are meaningful only if the database search pipeline computed them; if a pipeline outputs only raw scores without statistical tests, comparison requires separate statistical modeling (e.g., permutation or null-model fitting).70- Cross-pipeline comparison assumes all tools searched the same spectrum file set and the same (or comparable) chemical structure database; mismatches in database version or content will confound interpretation of tool-specific vs. true differences.71- Scan identifier matching across tool outputs requires careful attention to format and naming conventions; if pipelines report scan IDs differently (e.g., with or without file prefixes), a mapping step is necessary before tabulation.72- False discovery rate control is valid only if applied uniformly across all comparisons; if tools use different FDR estimation methods or null models, direct FDR comparison may be misleading.73- Mode-dependent sensitivity assessment (e.g., standard vs. blind MetaMiner) is only informative if both modes are run on identical input spectra and with matched search parameters; parameter differences can dominate mode effects.7475## Evidence7677- [methods] Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results: "Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results."78- [methods] Execute VarQuest pipeline on the same inputs with standard parameters: "Execute VarQuest pipeline on the same inputs with standard parameters."79- [methods] Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates: "Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates."80- [methods] Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences: "Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences."81- [readme] Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra: "Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra"82- [readme] VarQuest — a tool for modification-tolerant identification of novel variants of PNPs: "VarQuest — a tool for modification-tolerant identification of novel variants of PNPs"83- [readme] Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra: "Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra"
HolobiomicsLab