spectral-metadata-grouping
License: noncommercial — confirm your use is a permitted (noncommercial) purpose before applying; commercial use requires a separate license (see metadata.tool_license).
Summary
Group mass spectra by identical SPLASH keys to identify and remove duplicate spectral records in standardized spectral datasets. This skill enables systematic deduplication of mass spectrometry libraries by leveraging the SPLASH (SPectraL hASH) identifier—a database-independent, unambiguous spectral hash that serves as a unique fingerprint for each spectrum.
When to use
Apply this skill when processing a mass spectrometry dataset (in FragHub JSON format or similar) where duplicate spectral records are suspected or known to exist. The input dataset should already be in a standardized format with computed or retrievable SPLASH keys. Use this skill as a quality control step before final library assembly or when merging multiple spectral databases where redundancy is likely.
When NOT to use
- Input dataset already has known-unique spectra or has been pre-deduplicated by another method (e.g., exact m/z-intensity matching or database ID uniqueness).
- SPLASH keys are unavailable or cannot be computed (e.g., spectra lack m/z-intensity data or are in non-standard formats that spectra-hash does not support).
- The analysis goal requires retaining all variant forms of a spectrum (e.g., multiple collision energies, different instruments, or intentional replicates for meta-analysis)—in which case a different grouping strategy (e.g., by precursor m/z + adduct only) may be more appropriate.
Inputs
- Mass spectrometry spectral dataset (FragHub JSON format, CSV, MSP, or MGF)
- Per-spectrum SPLASH key (computed via spectra-hash or pre-existing field)
- Ion m/z and intensity pairs for each spectrum
Outputs
- Deduplicated spectral dataset in FragHub JSON format (or original format)
- Set of retained representative spectra (one per unique SPLASH key)
- Optionally: log or metadata file listing removed duplicate spectra and their corresponding SPLASH keys
How to apply
Load the spectral dataset into memory using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool (available as a Python module or command-line utility). Group all spectra by their SPLASH key value, treating identical keys as equivalence classes. Within each group, retain a single representative spectrum (e.g., the first occurrence, or the spectrum with the highest spectral quality metric if available) and mark all others as duplicates. Write the deduplicated spectral set to output, maintaining the original JSON structure and preserving all metadata fields for retained spectra. Verify deduplication by confirming that the output dataset has fewer or equal spectra than the input, and that no two spectra in the output share the same SPLASH key.
Related tools
- spectra-hash (Computes or validates SPLASH keys for each spectrum; provides Python, C++, Java, C#, and R implementations; used to generate the spectral hash identifier used as the grouping key.) — https://github.com/berlinguyinca/spectra-hash
- FragHub (Host application for standardizing and organizing mass spectrometry data; integrates spectra-hash directly for deduplication; orchestrates the full workflow from data loading to deduplicated output in standardized JSON format.) — https://github.com/eMetaboHUB/FragHub
- Python 3.12 (Programming environment for implementing the grouping and deduplication workflow; used to load, parse, group, and write spectral datasets.)
Evaluation signals
- Output spectral count is less than or equal to input spectral count; the difference equals the number of removed duplicates.
- No two spectra in the deduplicated output share the same SPLASH key (uniqueness invariant).
- All retained spectra maintain their original metadata fields and JSON structure integrity (no data loss for non-duplicate records).
- Each removed duplicate is correctly paired with an existing retained spectrum that has an identical SPLASH key.
- Spot-check: manually verify that a sample of spectra marked as duplicates do indeed have identical or near-identical m/z and intensity patterns when compared to their representative spectrum.
Limitations
- SPLASH key computation depends on the quality and completeness of m/z and intensity data; spectra with missing or malformed peak lists may fail to generate valid SPLASH keys or may produce spurious matches.
- SPLASH is designed to hash spectral profile (m/z and intensity) but does not account for metadata differences (e.g., collision energy, instrument, acquisition date); two physically different experiments may yield identical SPLASH keys if their peak profiles are the same, potentially causing over-deduplication in multi-source datasets.
- The choice of which duplicate to retain (first occurrence vs. highest quality) is not determined by SPLASH; a heuristic or explicit quality metric must be supplied externally.
- Large datasets (millions of spectra) may require efficient grouping and memory management; the provided workflow does not address multi-threading or chunking strategies mentioned in FragHub discussion.
- Spectra lacking SMILES, InChI, and InChIKey may be removed by FragHub before deduplication is applied, reducing the effective dataset size; deduplication should be performed on the full input dataset prior to such filtering if retention of all chemically distinct spectra is desired.
Evidence
- [other] FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset.: "FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset."
- [other] Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal.: "Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal."
- [readme] The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures.: "The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures."
- [other] Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool.: "Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool."
- [other] direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub.: "direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub."
- [other] Refactoring duplicatas removal, now by sames SPLASH key.: "Refactoring duplicatas removal, now by sames SPLASH key."
1---2name: spectral-metadata-grouping3description: Use when processing a mass spectrometry dataset (in FragHub JSON format or similar) where duplicate spectral records are suspected or known to exist. The input dataset should already be in a standardized format with computed or retrievable SPLASH keys.4license: CC-BY-4.05---67# spectral-metadata-grouping89> **License: noncommercial** — confirm your use is a permitted (noncommercial) purpose before applying; commercial use requires a separate license (see `metadata.tool_license`). <!-- asb-license-banner -->10## Summary1112Group mass spectra by identical SPLASH keys to identify and remove duplicate spectral records in standardized spectral datasets. This skill enables systematic deduplication of mass spectrometry libraries by leveraging the SPLASH (SPectraL hASH) identifier—a database-independent, unambiguous spectral hash that serves as a unique fingerprint for each spectrum.1314## When to use1516Apply this skill when processing a mass spectrometry dataset (in FragHub JSON format or similar) where duplicate spectral records are suspected or known to exist. The input dataset should already be in a standardized format with computed or retrievable SPLASH keys. Use this skill as a quality control step before final library assembly or when merging multiple spectral databases where redundancy is likely.1718## When NOT to use1920- Input dataset already has known-unique spectra or has been pre-deduplicated by another method (e.g., exact m/z-intensity matching or database ID uniqueness).21- SPLASH keys are unavailable or cannot be computed (e.g., spectra lack m/z-intensity data or are in non-standard formats that spectra-hash does not support).22- The analysis goal requires retaining all variant forms of a spectrum (e.g., multiple collision energies, different instruments, or intentional replicates for meta-analysis)—in which case a different grouping strategy (e.g., by precursor m/z + adduct only) may be more appropriate.2324## Inputs2526- Mass spectrometry spectral dataset (FragHub JSON format, CSV, MSP, or MGF)27- Per-spectrum SPLASH key (computed via spectra-hash or pre-existing field)28- Ion m/z and intensity pairs for each spectrum2930## Outputs3132- Deduplicated spectral dataset in FragHub JSON format (or original format)33- Set of retained representative spectra (one per unique SPLASH key)34- Optionally: log or metadata file listing removed duplicate spectra and their corresponding SPLASH keys3536## How to apply3738Load the spectral dataset into memory using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool (available as a Python module or command-line utility). Group all spectra by their SPLASH key value, treating identical keys as equivalence classes. Within each group, retain a single representative spectrum (e.g., the first occurrence, or the spectrum with the highest spectral quality metric if available) and mark all others as duplicates. Write the deduplicated spectral set to output, maintaining the original JSON structure and preserving all metadata fields for retained spectra. Verify deduplication by confirming that the output dataset has fewer or equal spectra than the input, and that no two spectra in the output share the same SPLASH key.3940## Related tools4142- **spectra-hash** (Computes or validates SPLASH keys for each spectrum; provides Python, C++, Java, C#, and R implementations; used to generate the spectral hash identifier used as the grouping key.) — https://github.com/berlinguyinca/spectra-hash43- **FragHub** (Host application for standardizing and organizing mass spectrometry data; integrates spectra-hash directly for deduplication; orchestrates the full workflow from data loading to deduplicated output in standardized JSON format.) — https://github.com/eMetaboHUB/FragHub44- **Python 3.12** (Programming environment for implementing the grouping and deduplication workflow; used to load, parse, group, and write spectral datasets.)4546## Evaluation signals4748- Output spectral count is less than or equal to input spectral count; the difference equals the number of removed duplicates.49- No two spectra in the deduplicated output share the same SPLASH key (uniqueness invariant).50- All retained spectra maintain their original metadata fields and JSON structure integrity (no data loss for non-duplicate records).51- Each removed duplicate is correctly paired with an existing retained spectrum that has an identical SPLASH key.52- Spot-check: manually verify that a sample of spectra marked as duplicates do indeed have identical or near-identical m/z and intensity patterns when compared to their representative spectrum.5354## Limitations5556- SPLASH key computation depends on the quality and completeness of m/z and intensity data; spectra with missing or malformed peak lists may fail to generate valid SPLASH keys or may produce spurious matches.57- SPLASH is designed to hash spectral *profile* (m/z and intensity) but does not account for metadata differences (e.g., collision energy, instrument, acquisition date); two physically different experiments may yield identical SPLASH keys if their peak profiles are the same, potentially causing over-deduplication in multi-source datasets.58- The choice of which duplicate to retain (first occurrence vs. highest quality) is not determined by SPLASH; a heuristic or explicit quality metric must be supplied externally.59- Large datasets (millions of spectra) may require efficient grouping and memory management; the provided workflow does not address multi-threading or chunking strategies mentioned in FragHub discussion.60- Spectra lacking SMILES, InChI, and InChIKey may be removed by FragHub *before* deduplication is applied, reducing the effective dataset size; deduplication should be performed on the full input dataset prior to such filtering if retention of all chemically distinct spectra is desired.6162## Evidence6364- [other] FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset.: "FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset."65- [other] Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal.: "Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal."66- [readme] The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures.: "The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures."67- [other] Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool.: "Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool."68- [other] direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub.: "direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub."69- [other] Refactoring duplicatas removal, now by sames SPLASH key.: "Refactoring duplicatas removal, now by sames SPLASH key."
HolobiomicsLab