Spectral Scoring And Significance Assessment

spectral-scoring-and-significance-assessment

Summary

Evaluate and compare database search results from natural product mass spectrometry pipelines by computing match scores, p-values, and false discovery rates (FDR) to distinguish true identifications from noise. This skill is essential for validating which database search pipeline (Dereplicator, VarQuest, Dereplicator+) provides reliable identifications and assessing the statistical confidence of spectrum-to-structure matches.

When to use

You have executed multiple database search pipelines (Dereplicator, VarQuest, and/or Dereplicator+) on centroided LC-MS/MS spectra (in MGF, mzXML, mzML, or mzData format) against a chemical structure database of known natural products, and you need to: (1) identify which pipeline(s) successfully detected matches in your test spectra, (2) rank matches by statistical significance rather than raw score alone, (3) cross-compare hit patterns and sensitivity across search modes (e.g., standard vs. blind/unbiased modification search), or (4) control false discovery rate when reporting identifications.

When NOT to use

• Spectra have not yet been searched against a database (this skill assumes search results already exist; it is a post-processing step, not a search tool itself).
• You are using only a single database search pipeline and have no cross-pipeline comparison goal — in that case, focus on single-tool result interpretation rather than comparative analysis.
• Raw, uncentroided spectra or spectra in proprietary vendor formats that have not been converted to MGF/mzXML/mzML — preprocessing and format conversion must precede this skill.

Inputs

• significant_matches.tsv output from Dereplicator pipeline
• significant_matches.tsv output from VarQuest pipeline
• significant_matches.tsv output from Dereplicator+ pipeline
• Scan identifiers from test spectra (MSV000080102 .mzML files converted to MGF)
• Match scores (numeric)
• P-values (numeric, one per match)
• False discovery rates (numeric, one per match)

Outputs

• Cross-compared hit set table (scan ID, pipeline, score, p-value, FDR, common vs. unique)
• Tool-specific detections list (matches found by only one pipeline)
• Common identifications list (matches found by ≥2 pipelines)
• Mode-dependent sensitivity analysis (standard vs. blind/modification-tolerant mode comparison)
• Ranked match list filtered by p-value and FDR thresholds

How to apply

Parse the significant_matches.tsv output files generated by each database search tool, extracting scan identifiers, match scores, p-values, and false discovery rates for each spectrum–structure pair. Tabulate these metrics side-by-side to enable tool-to-tool comparison. Filter matches using both score and statistical thresholds (p-value and FDR cutoffs should be justified based on your false positive tolerance). Identify tool-specific detections (matches unique to one pipeline), common identifications (hits shared across pipelines, which increase confidence), and mode-dependent sensitivity differences (e.g., standard Dereplicator vs. blind-mode MetaMiner). Cross-reference scan identifiers to ensure you are comparing the same spectra across tools. Use p-values and FDR to rank and report matches in order of statistical confidence rather than raw score alone, as this guards against spurious high-scoring alignments.

Related tools

• Dereplicator (database search pipeline for peptidic natural products; generates match scores and results for tabulation and cross-comparison) — https://github.com/ablab/npdtools
• VarQuest (modification-tolerant database search pipeline; produces alternative hits and sensitivity profile for comparison with Dereplicator and Dereplicator+) — https://github.com/ablab/npdtools
• Dereplicator+ (extended database search pipeline for both peptidic and non-peptidic metabolites; provides an alternative hit set for cross-validation) — https://github.com/ablab/npdtools
• MetaMiner (metabologenomic pipeline that includes database search via Dereplicator; supports standard and blind (unbiased modification) search modes for mode-dependent sensitivity comparison) — https://github.com/ablab/npdtools
• Python (scripting language used to parse significant_matches.tsv, tabulate results, and cross-compare metrics across tools)

Examples

python parse_matches.py --dereplicator significant_matches.tsv --varquest significant_matches.tsv --dereplicate_plus significant_matches.tsv --output cross_comparison.tsv

Evaluation signals

• Cross-comparison table is complete: every scan ID from the test spectra appears in the output with entries for all pipelines that detected a match (or an explicit 'no match' entry).
• P-values and FDR values are present and finite (not NaN or infinite) for all reported matches; FDR is monotonically non-decreasing when matches are sorted by p-value within each tool.
• Tool-specific and common identification lists are mutually exclusive and exhaustive (every match belongs to exactly one category: unique to one tool, or shared by ≥2 tools).
• Matches ranked by p-value show expected rank order (lower p-value = more confident match); matches with identical p-values can be sub-ranked by score if reported.
• Mode-dependent comparison (standard vs. blind search) shows that blind/modification-tolerant modes detect additional matches not found in standard mode, validating the added sensitivity of flexible search.

Limitations

• P-values and FDR are meaningful only if the database search pipeline computed them; if a pipeline outputs only raw scores without statistical tests, comparison requires separate statistical modeling (e.g., permutation or null-model fitting).
• Cross-pipeline comparison assumes all tools searched the same spectrum file set and the same (or comparable) chemical structure database; mismatches in database version or content will confound interpretation of tool-specific vs. true differences.
• Scan identifier matching across tool outputs requires careful attention to format and naming conventions; if pipelines report scan IDs differently (e.g., with or without file prefixes), a mapping step is necessary before tabulation.
• False discovery rate control is valid only if applied uniformly across all comparisons; if tools use different FDR estimation methods or null models, direct FDR comparison may be misleading.
• Mode-dependent sensitivity assessment (e.g., standard vs. blind MetaMiner) is only informative if both modes are run on identical input spectra and with matched search parameters; parameter differences can dominate mode effects.

Evidence

• [methods] Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results: "Execute Dereplicator pipeline on the same test spectra and RiPP structure database, output match results."
• [methods] Execute VarQuest pipeline on the same inputs with standard parameters: "Execute VarQuest pipeline on the same inputs with standard parameters."
• [methods] Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates: "Parse and tabulate significant_matches.tsv outputs from all three tools, recording scan identifiers, match scores, p-values, and false discovery rates."
• [methods] Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences: "Cross-compare hit sets to identify tool-specific detections, common identifications, and mode-dependent sensitivity differences."
• [readme] Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra: "Dereplicator — a tool for identification of peptidic natural products (PNPs) through database search of mass spectra"
• [readme] VarQuest — a tool for modification-tolerant identification of novel variants of PNPs: "VarQuest — a tool for modification-tolerant identification of novel variants of PNPs"
• [readme] Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra: "Dereplicator+ — a tool for identification of metabolites (both peptidic and non-peptidic) through database search of mass spectra"