hash-based-deduplication-workflow
License: noncommercial — confirm your use is a permitted (noncommercial) purpose before applying; commercial use requires a separate license (see metadata.tool_license).
Summary
A data quality workflow that identifies and removes duplicate mass spectra by computing spectral hash identifiers (SPLASH keys) and grouping spectra by identical hash values. This ensures a deduplicated spectral dataset while preserving the first occurrence of each unique spectrum.
When to use
When processing open mass spectrometry library (OMSL) data that may contain duplicate spectral records (e.g., the same compound measured under identical conditions), you need to reduce redundancy while retaining one representative spectrum per unique spectral profile to improve downstream analysis efficiency and avoid biasing database search results.
When NOT to use
- Input spectra lack complete chemical structure information (SMILES, InChI, AND InChIKey) — these should be filtered out before deduplication
- Spectra have missing or ambiguous precursor m/z and adduct annotations — address adduct validation before hash-based grouping
- Intentional spectrum replicates are part of the study design (e.g., technical replicates for quality metrics) — in this case, preserve replicates and mark them explicitly rather than deduplicating
Inputs
- Spectral dataset in FragHub JSON format
- Mass spectrometry records with m/z and intensity pairs
- Complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct information)
Outputs
- Deduplicated spectral dataset in FragHub JSON format
- Deletion log documenting removed duplicate spectra and their SPLASH keys
- Single representative spectrum per unique SPLASH key retained in output
How to apply
Load the spectral dataset in FragHub JSON format using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool, which generates an unambiguous, database-independent identifier based on the m/z and intensity pairs in the spectrum. Group spectra by identical SPLASH key values; spectra with the same SPLASH key are exact duplicates. For each group of identical SPLASH keys, retain only the first occurrence (or another deterministic representative) and mark all others for removal. Write the deduplicated spectral set to output, preserving the original JSON structure. The SPLASH identifier has four blocks separated by dashes (e.g., splash10-0002-0900000000-b112e4e059e1ecf98c5f), with the fourth block containing the unique hash that enables duplicate detection.
Related tools
- spectra-hash (Computes SPLASH keys for each spectrum; core tool for spectral hashing and grouping by identical hash values) — https://github.com/berlinguyinca/spectra-hash
- Python 3.12 (Execution environment for loading spectral dataset, orchestrating SPLASH computation, grouping by hash key, and writing deduplicated output)
- FragHub (Integrates spectra-hash; handles spectral dataset standardization, filtering, and deduplication workflow) — https://github.com/eMetaboHUB/FragHub
Evaluation signals
- Total number of input spectra vs. deduplicated output spectra — verify a reduction in dataset size consistent with the expected duplicate rate
- All retained spectra have unique SPLASH keys — check that no two spectra in the output share the same SPLASH key
- First-occurrence preservation — confirm that the retained representative for each group is deterministically selected (e.g., earliest in file order)
- Deletion log completeness — verify that all removed spectra are logged with their SPLASH key, original identifier, and reason for removal
- JSON schema validity — validate that output spectra retain original JSON structure and all metadata fields (m/z, intensity, SMILES, InChI, InChIKey, precursor m/z, adduct)
Limitations
- SPLASH keys are computed from m/z and intensity pairs only; minor spectral variations (e.g., different ionization energies, slight peak shifts due to instrument calibration drift) may produce different SPLASH keys for chemically identical spectra, resulting in missed duplicates
- Duplicate detection is exact-match based on SPLASH hash — near-duplicates or spectra with minor peak differences will not be flagged
- Requires complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct); spectra lacking any of these are removed before deduplication occurs, potentially discarding useful spectral data
- FragHub has removed macOS and Linux support as of version 1.4.1, restricting execution to Windows x64
Evidence
- [other] FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset.: "FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset."
- [other] Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal.: "Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values."
- [readme] The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate blocks that define different layers of information, separated by dashes.: "The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate"
- [readme] All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub.: "All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub."
- [other] direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub.: "direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub."
1---2name: hash-based-deduplication-workflow3description: Use when when processing open mass spectrometry library (OMSL) data that may contain duplicate spectral records (e.4license: CC-BY-4.05---67# hash-based-deduplication-workflow89> **License: noncommercial** — confirm your use is a permitted (noncommercial) purpose before applying; commercial use requires a separate license (see `metadata.tool_license`). <!-- asb-license-banner -->10## Summary1112A data quality workflow that identifies and removes duplicate mass spectra by computing spectral hash identifiers (SPLASH keys) and grouping spectra by identical hash values. This ensures a deduplicated spectral dataset while preserving the first occurrence of each unique spectrum.1314## When to use1516When processing open mass spectrometry library (OMSL) data that may contain duplicate spectral records (e.g., the same compound measured under identical conditions), you need to reduce redundancy while retaining one representative spectrum per unique spectral profile to improve downstream analysis efficiency and avoid biasing database search results.1718## When NOT to use1920- Input spectra lack complete chemical structure information (SMILES, InChI, AND InChIKey) — these should be filtered out before deduplication21- Spectra have missing or ambiguous precursor m/z and adduct annotations — address adduct validation before hash-based grouping22- Intentional spectrum replicates are part of the study design (e.g., technical replicates for quality metrics) — in this case, preserve replicates and mark them explicitly rather than deduplicating2324## Inputs2526- Spectral dataset in FragHub JSON format27- Mass spectrometry records with m/z and intensity pairs28- Complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct information)2930## Outputs3132- Deduplicated spectral dataset in FragHub JSON format33- Deletion log documenting removed duplicate spectra and their SPLASH keys34- Single representative spectrum per unique SPLASH key retained in output3536## How to apply3738Load the spectral dataset in FragHub JSON format using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool, which generates an unambiguous, database-independent identifier based on the m/z and intensity pairs in the spectrum. Group spectra by identical SPLASH key values; spectra with the same SPLASH key are exact duplicates. For each group of identical SPLASH keys, retain only the first occurrence (or another deterministic representative) and mark all others for removal. Write the deduplicated spectral set to output, preserving the original JSON structure. The SPLASH identifier has four blocks separated by dashes (e.g., `splash10-0002-0900000000-b112e4e059e1ecf98c5f`), with the fourth block containing the unique hash that enables duplicate detection.3940## Related tools4142- **spectra-hash** (Computes SPLASH keys for each spectrum; core tool for spectral hashing and grouping by identical hash values) — https://github.com/berlinguyinca/spectra-hash43- **Python 3.12** (Execution environment for loading spectral dataset, orchestrating SPLASH computation, grouping by hash key, and writing deduplicated output)44- **FragHub** (Integrates spectra-hash; handles spectral dataset standardization, filtering, and deduplication workflow) — https://github.com/eMetaboHUB/FragHub4546## Evaluation signals4748- Total number of input spectra vs. deduplicated output spectra — verify a reduction in dataset size consistent with the expected duplicate rate49- All retained spectra have unique SPLASH keys — check that no two spectra in the output share the same SPLASH key50- First-occurrence preservation — confirm that the retained representative for each group is deterministically selected (e.g., earliest in file order)51- Deletion log completeness — verify that all removed spectra are logged with their SPLASH key, original identifier, and reason for removal52- JSON schema validity — validate that output spectra retain original JSON structure and all metadata fields (m/z, intensity, SMILES, InChI, InChIKey, precursor m/z, adduct)5354## Limitations5556- SPLASH keys are computed from m/z and intensity pairs only; minor spectral variations (e.g., different ionization energies, slight peak shifts due to instrument calibration drift) may produce different SPLASH keys for chemically identical spectra, resulting in missed duplicates57- Duplicate detection is exact-match based on SPLASH hash — near-duplicates or spectra with minor peak differences will not be flagged58- Requires complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct); spectra lacking any of these are removed before deduplication occurs, potentially discarding useful spectral data59- FragHub has removed macOS and Linux support as of version 1.4.1, restricting execution to Windows x646061## Evidence6263- [other] FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset.: "FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset."64- [other] Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal.: "Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values."65- [readme] The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate blocks that define different layers of information, separated by dashes.: "The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate"66- [readme] All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub.: "All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub."67- [other] direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub.: "direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub."
HolobiomicsLab 