Hit Score Consolidation Across Domains

hit-score-consolidation-across-domains

Summary

Merge and rank mass spectrometry spectral library matches across multiple domain-specific MASST search results (microbe, plant, tissue, microbiome, food) into a single consolidated output with unified scoring and metadata. This skill enables cross-domain comparison and interpretation of spectral similarity hits without redundancy or domain-specific bias.

When to use

You have executed batch spectral searches against two or more domain-specific MASSTs (via the Fast Search API or individual MASST tools) and need to combine the resulting hit lists, match scores, and taxonomic/metadata annotations into a single ranked table for downstream analysis, visualization, or Level 2 metabolite annotation. Use this when your unknown spectrum(s) may match organisms or samples spanning multiple biological domains and you want to avoid reporting the same library match multiple times.

When NOT to use

• Single-spectrum, single-domain searches: use individual MASST tools directly without consolidation.
• Domain-specific mechanistic analysis: if your study focus is microbial metabolism only, consolidating across plant/tissue domains introduces unneeded noise.
• Pre-consolidated outputs: if your pipeline already merges hits upstream (e.g., in a custom workflow), re-consolidation risks double-deduplication or score inflation.

Inputs

• _matches.tsv file (all scans matching searched spectrum across indexed data)
• _library.tsv file (GNPS library hits for Level 2 annotation)
• _count_domain.tsv files (per-domain match counts and metadata)
• _datasets.tsv file (unique sample counts per indexed dataset)
• Batch search parameters (cosine threshold, mz tolerance, minimum matching peaks)

Outputs

• Consolidated hit table (TSV) with deduplicated matches, merged scores, and domain annotations
• Unified library annotation table with cross-domain metadata
• Domain overlap summary (spectra appearing in multiple MASSTs)
• Ranked match list by consolidated cosine score and domain breadth
• Structured input for metadataMASST visualization (optional)

How to apply

Load the _matches.tsv, _library.tsv, and _count_domain.tsv outputs from each domain-specific MASST batch search (generated by jobs.py). Consolidate matches by deduplicating on USI or spectrum identifier, retaining the highest cosine score and the union of domain assignments for each unique hit. Merge metadata fields (organism lineage, tissue type, sample origin) across domain sources to create a master annotation table. Apply consistent thresholds for minimum cosine similarity (typically ≥ 0.5–0.7, tunable via jobs.py parameters) and minimum matching peaks to filter low-confidence hits before consolidation. Sort the consolidated table by descending cosine score, then by count across domains, to surface hits that appear in multiple MASSTs or have strong scores. Optionally feed consolidated output to metadataMASST for visualization of domain overlap and hit distribution.

Related tools

• jobs.py (Executes batch spectral search against multiple domain-specific MASSTs via Fast Search API; generates per-domain TSV and JSON outputs to be consolidated) — https://github.com/robinschmid/microbe_masst
• metadataMASST (Accepts consolidated hit output to generate interactive visualization of aggregated results, domain overlap, and score rankings) — https://masst.gnps2.org/metadatamasst/
• microbeMASST (Domain-specific search tool (microbes); contributes _microbe.tsv matches to be consolidated with other domains) — https://masst.gnps2.org/microbemasst/
• plantMASST (Domain-specific search tool (plants); contributes _plant.tsv matches to consolidation pool) — https://masst.gnps2.org/plantmasst/
• tissueMASST (Domain-specific search tool (tissues); contributes _tissue.tsv matches to consolidation pool) — https://masst.gnps2.org/tissuemasst/
• microbiomeMASST (Domain-specific search tool (microbiomes); contributes _microbiome.tsv matches to consolidation pool) — https://masst.gnps2.org/microbiomemasst/
• foodMASST (Domain-specific search tool (food samples); contributes _food.tsv matches to consolidation pool) — https://masst.gnps2.org/foodmasst2/
• Fast Search API (Backend search engine called by jobs.py to query indexed data from GNPS/MassIVE, Metabolomics Workbench, Metabolights, and NORMAN; enables batch processing) — https://fasst.gnps2.org/fastsearch/
• GNPS_MASST (Codebase containing standalone web application code for all domain-specific MASSTs; underlying implementation for consolidation-ready outputs) — https://github.com/mwang87/GNPS_MASST

Examples

python jobs.py  # (after editing files list and parameters in jobs.py); then merge output _matches.tsv and _count_domain.tsv files using: `awk 'FNR==1 && NR>1 {next} {print}' *.matches.tsv | sort -t$'\t' -k3 -rn > consolidated_hits.tsv`

Evaluation signals

• No duplicate USI/spectrum identifiers in consolidated output; each unique match appears exactly once with maximum cosine score retained.
• All domain assignments for a given hit are present in a single row (e.g., 'microbe,plant' if hit appears in both MASST searches); verify union logic.
• Consolidated cosine scores are monotonically non-increasing when sorted descending; verify no score inflation or data corruption during merge.
• Domain count column matches length of comma-separated domain list; spot-check 5–10 rows for consistency.
• Library vs. non-library hits are distinguishable (e.g., separate columns or sections in output); Level 2 vs. non-annotated hits must be traceable.
• Consolidated output is valid input to metadataMASST visualization without schema errors or missing required fields.

Limitations

• Score comparability across domains: cosine similarity thresholds and indexed data sizes differ per MASST; consolidation does not normalize for domain-specific score distributions. Users should examine per-domain score distributions or apply domain-specific thresholds before consolidation if mechanistic conclusions are domain-sensitive.
• API failure resilience: the README notes that Fast Search API requests may fail on first attempt; jobs.py mitigates this via re-runs (skip_existing=True), but consolidation must handle partial outputs (missing _count_domain.tsv files from failed runs).
• Python version dependency: jobs.py requires Python 3.10 specifically; consolidation scripts using the same environment may be incompatible with Python 3.11+ or 3.9, limiting reproducibility across compute environments.
• Indexed data staleness: consolidation reflects only currently indexed data in GNPS/MassIVE, Metabolomics Workbench, Metabolights, and NORMAN; new library data or curations in any MASST are not retroactively applied to prior consolidated outputs.
• Lineage coverage gaps: plantMASST and microbeMASST lineage tables show incomplete species/strain coverage (e.g., only 3712 plant species vs. millions known); hits to underrepresented taxa may not be consolidated with taxonomic confidence.

Evidence

• [other] Process and consolidate search results: "Process and consolidate search results using metadataMASST to merge hits, scores, and metadata across domain sources."
• [readme] Batch search and multi-domain output generation: "Running jobs.py allows users to leverage the Fast Search API and execute a batch search"
• [readme] Consolidated output file descriptions: "A _matches.tsv file will be generated. This contains all the scans found to match your searched spectrum of interest in the data that have been currently indexed."
• [intro] Cross-domain match aggregation into visualizations: "Aggregated search outputs can be generated and visualized using metadataMASST"
• [readme] Domain-specific MASST ecosystem scope: "This repository contains the code and data for the different domain-specific MASSTs currently under development in the Dorrestein Lab at UC San Diego. This includes microbeMASST, plantMASST,"
• [readme] Parameter tuning for batch searches: "Check and adjust the different parameters for the search, such as minimum cosine score, mz tolerance, and number of minimum matching peaks based on your research question."