Cross Domain Masst Output Parsing

cross-domain-masst-output-parsing

Summary

Parse and normalize search result files from multiple domain-specific MASST tools (microbeMASST, plantMASST, tissueMASST, microbiomeMASST, foodMASST) into a unified table, extracting spectrum identifiers, match scores, and metadata fields for aggregation and visualization via metadataMASST.

When to use

You have executed batch searches against one or more domain-specific MASST tools and received multiple output files (_microbe.html, _plant.html, _tissue.html, _microbiome.html, _food.html, _matches.tsv, _library.tsv, _datasets.tsv, count*.tsv) that must be combined into a single normalized summary for cross-domain comparison or metadataMASST web interface visualization.

When NOT to use

• Input is a single domain-specific MASST output file; use domain-specific visualization directly rather than aggregation.
• Search has not yet been executed; first run the batch search via jobs.py or the Fast Search API.
• Output files are from different batch runs or incompatible software versions; ensure consistent provenance and schema.

Inputs

• domain-specific MASST output files (_microbe.tsv, _plant.tsv, _tissue.tsv, _microbiome.tsv, _food.tsv)
• spectrum match tables (_matches.tsv)
• library annotation tables (_library.tsv)
• dataset summary tables (_datasets.tsv)
• domain-specific count tables (count*.tsv)
• JSON tree structures (_microbe.json, etc.)

Outputs

• unified aggregated match table (TSV or JSON)
• normalized metadata-enriched summary artifact
• metadataMASST-compatible input file
• validation report (row counts, field completeness, score ranges)

How to apply

Load the output files generated by each domain-specific MASST run (typically .tsv, .json, or .html formats from the Fast Search API). Identify and extract key fields from each output: spectrum identifiers, cosine similarity scores, dataset/sample metadata, and domain-specific lineage or taxonomy information. Normalize field names and data types across domain sources to ensure consistent column names and value ranges (e.g., all cosine scores to 0–1 float format). Merge records across all domain outputs into a unified table by spectrum identifier, preserving source domain attribution. Validate aggregated output by checking row counts match input cardinality, examining score distributions for expected ranges, and confirming no required metadata fields are null. Generate a final aggregated table or JSON structure suitable for import into the metadataMASST visualization interface.

Related tools

• metadataMASST (target visualization platform; accepts aggregated search outputs and generates interactive summary artifacts) — https://masst.gnps2.org/metadatamasst/
• microbeMASST (source domain-specific MASST tool; generates microbial search outputs to be aggregated) — https://github.com/robinschmid/microbe_masst
• plantMASST (source domain-specific MASST tool; generates plant search outputs to be aggregated)
• tissueMASST (source domain-specific MASST tool; generates tissue search outputs to be aggregated)
• microbiomeMASST (source domain-specific MASST tool; generates microbiome search outputs to be aggregated)
• foodMASST (source domain-specific MASST tool; generates food search outputs to be aggregated)
• jobs.py (batch search orchestration; executes searches against all domainMASSTs and generates the output files to be parsed) — https://github.com/robinschmid/microbe_masst/blob/master/code/jobs.py
• GNPS_MASST (code repository containing underlying web application logic for standalone domain-specific MASST tools) — https://github.com/mwang87/GNPS_MASST

Evaluation signals

• Aggregated table row count equals the sum of input file row counts (accounting for deduplication by spectrum ID if applicable).
• No required metadata columns are entirely null; spot-check a sample of 5–10 records across each source domain for completeness.
• Cosine similarity scores fall within the expected range [0, 1] and distribution matches the input output ranges from each domain.
• Spectrum identifiers are unique (no duplicates) and match USI or internal GNPS/MassIVE naming conventions.
• The final artifact is loadable by the metadataMASST web interface without parsing or schema errors.

Limitations

• Some entries may fail during Fast Search API calls; re-running jobs.py multiple times with skip_existing=True is required to ensure complete coverage.
• The Fast Search API indexes GNPS/MassIVE, Metabolomics Workbench, Metabolights, and NORMAN data only; spectra outside these repositories will not match.
• Different domain-specific MASSTs have different lineage coverage (e.g., microbeMASST covers 1379 species, plantMASST covers 3712 species); aggregation does not fill gaps in sparse domains.
• Python 3.10 is required for the jobs.py execution; version mismatches may cause pipeline failures.

Evidence

• [other] Load search output files from one or more domain-specific MASST tools: "Load search output files from one or more domain-specific MASST tools (microbeMASST, plantMASST, tissueMASST, microbiomeMASST, foodMASST)."
• [other] Parse and normalize search results across domains: "Parse and normalize search results, extracting spectrum identifiers, match scores, and metadata fields across domain sources."
• [other] Merge results into unified format: "Aggregate results by merging records from multiple MASST outputs into a unified table or structured format."
• [other] Generate visualization artifacts: "Generate a combined visualizable summary artifact compatible with the metadataMASST web interface."
• [other] Validate aggregated output: "Validate the aggregated output for completeness and consistency across input sources."
• [readme] Output file types from batch search: "A series of interactive HTML trees files will be generated for each domain-specific MASST ending with _domain.html (e.g., _microbe.html). A series of JSON files for the different trees will be"
• [readme] Aggregated search outputs visualized via metadataMASST: "Aggregated search outputs can be generated and visualized using metadataMASST"
• [readme] Re-running for completeness: "Make user to use Python 3.10. Due to the Fast Search API some of the entries will fail. Nevertheless sequent re-runs should catch all the possible matches."