hash-based-deduplication-workflow
Summary
A data quality workflow that identifies and removes duplicate mass spectra by computing spectral hash identifiers (SPLASH keys) and grouping spectra by identical hash values. This ensures a deduplicated spectral dataset while preserving the first occurrence of each unique spectrum.
When to use
When processing open mass spectrometry library (OMSL) data that may contain duplicate spectral records (e.g., the same compound measured under identical conditions), you need to reduce redundancy while retaining one representative spectrum per unique spectral profile to improve downstream analysis efficiency and avoid biasing database search results.
When NOT to use
- Input spectra lack complete chemical structure information (SMILES, InChI, AND InChIKey) — these should be filtered out before deduplication
- Spectra have missing or ambiguous precursor m/z and adduct annotations — address adduct validation before hash-based grouping
- Intentional spectrum replicates are part of the study design (e.g., technical replicates for quality metrics) — in this case, preserve replicates and mark them explicitly rather than deduplicating
Inputs
- Spectral dataset in FragHub JSON format
- Mass spectrometry records with m/z and intensity pairs
- Complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct information)
Outputs
- Deduplicated spectral dataset in FragHub JSON format
- Deletion log documenting removed duplicate spectra and their SPLASH keys
- Single representative spectrum per unique SPLASH key retained in output
How to apply
Load the spectral dataset in FragHub JSON format using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool, which generates an unambiguous, database-independent identifier based on the m/z and intensity pairs in the spectrum. Group spectra by identical SPLASH key values; spectra with the same SPLASH key are exact duplicates. For each group of identical SPLASH keys, retain only the first occurrence (or another deterministic representative) and mark all others for removal. Write the deduplicated spectral set to output, preserving the original JSON structure. The SPLASH identifier has four blocks separated by dashes (e.g., splash10-0002-0900000000-b112e4e059e1ecf98c5f), with the fourth block containing the unique hash that enables duplicate detection.
Related tools
- spectra-hash (Computes SPLASH keys for each spectrum; core tool for spectral hashing and grouping by identical hash values) — https://github.com/berlinguyinca/spectra-hash
- Python 3.12 (Execution environment for loading spectral dataset, orchestrating SPLASH computation, grouping by hash key, and writing deduplicated output)
- FragHub (Integrates spectra-hash; handles spectral dataset standardization, filtering, and deduplication workflow) — https://github.com/eMetaboHUB/FragHub
Evaluation signals
- Total number of input spectra vs. deduplicated output spectra — verify a reduction in dataset size consistent with the expected duplicate rate
- All retained spectra have unique SPLASH keys — check that no two spectra in the output share the same SPLASH key
- First-occurrence preservation — confirm that the retained representative for each group is deterministically selected (e.g., earliest in file order)
- Deletion log completeness — verify that all removed spectra are logged with their SPLASH key, original identifier, and reason for removal
- JSON schema validity — validate that output spectra retain original JSON structure and all metadata fields (m/z, intensity, SMILES, InChI, InChIKey, precursor m/z, adduct)
Limitations
- SPLASH keys are computed from m/z and intensity pairs only; minor spectral variations (e.g., different ionization energies, slight peak shifts due to instrument calibration drift) may produce different SPLASH keys for chemically identical spectra, resulting in missed duplicates
- Duplicate detection is exact-match based on SPLASH hash — near-duplicates or spectra with minor peak differences will not be flagged
- Requires complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct); spectra lacking any of these are removed before deduplication occurs, potentially discarding useful spectral data
- FragHub has removed macOS and Linux support as of version 1.4.1, restricting execution to Windows x64
Evidence
- [other] FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset.: "FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset."
- [other] Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal.: "Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values."
- [readme] The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate blocks that define different layers of information, separated by dashes.: "The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate"
- [readme] All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub.: "All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub."
- [other] direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub.: "direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub."
1---2name: hash-based-deduplication-workflow-23description: Use when when processing open mass spectrometry library (OMSL) data that may contain duplicate spectral records (e.4license: CC-BY-4.05---67# hash-based-deduplication-workflow89## Summary1011A data quality workflow that identifies and removes duplicate mass spectra by computing spectral hash identifiers (SPLASH keys) and grouping spectra by identical hash values. This ensures a deduplicated spectral dataset while preserving the first occurrence of each unique spectrum.1213## When to use1415When processing open mass spectrometry library (OMSL) data that may contain duplicate spectral records (e.g., the same compound measured under identical conditions), you need to reduce redundancy while retaining one representative spectrum per unique spectral profile to improve downstream analysis efficiency and avoid biasing database search results.1617## When NOT to use1819- Input spectra lack complete chemical structure information (SMILES, InChI, AND InChIKey) — these should be filtered out before deduplication20- Spectra have missing or ambiguous precursor m/z and adduct annotations — address adduct validation before hash-based grouping21- Intentional spectrum replicates are part of the study design (e.g., technical replicates for quality metrics) — in this case, preserve replicates and mark them explicitly rather than deduplicating2223## Inputs2425- Spectral dataset in FragHub JSON format26- Mass spectrometry records with m/z and intensity pairs27- Complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct information)2829## Outputs3031- Deduplicated spectral dataset in FragHub JSON format32- Deletion log documenting removed duplicate spectra and their SPLASH keys33- Single representative spectrum per unique SPLASH key retained in output3435## How to apply3637Load the spectral dataset in FragHub JSON format using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool, which generates an unambiguous, database-independent identifier based on the m/z and intensity pairs in the spectrum. Group spectra by identical SPLASH key values; spectra with the same SPLASH key are exact duplicates. For each group of identical SPLASH keys, retain only the first occurrence (or another deterministic representative) and mark all others for removal. Write the deduplicated spectral set to output, preserving the original JSON structure. The SPLASH identifier has four blocks separated by dashes (e.g., `splash10-0002-0900000000-b112e4e059e1ecf98c5f`), with the fourth block containing the unique hash that enables duplicate detection.3839## Related tools4041- **spectra-hash** (Computes SPLASH keys for each spectrum; core tool for spectral hashing and grouping by identical hash values) — https://github.com/berlinguyinca/spectra-hash42- **Python 3.12** (Execution environment for loading spectral dataset, orchestrating SPLASH computation, grouping by hash key, and writing deduplicated output)43- **FragHub** (Integrates spectra-hash; handles spectral dataset standardization, filtering, and deduplication workflow) — https://github.com/eMetaboHUB/FragHub4445## Evaluation signals4647- Total number of input spectra vs. deduplicated output spectra — verify a reduction in dataset size consistent with the expected duplicate rate48- All retained spectra have unique SPLASH keys — check that no two spectra in the output share the same SPLASH key49- First-occurrence preservation — confirm that the retained representative for each group is deterministically selected (e.g., earliest in file order)50- Deletion log completeness — verify that all removed spectra are logged with their SPLASH key, original identifier, and reason for removal51- JSON schema validity — validate that output spectra retain original JSON structure and all metadata fields (m/z, intensity, SMILES, InChI, InChIKey, precursor m/z, adduct)5253## Limitations5455- SPLASH keys are computed from m/z and intensity pairs only; minor spectral variations (e.g., different ionization energies, slight peak shifts due to instrument calibration drift) may produce different SPLASH keys for chemically identical spectra, resulting in missed duplicates56- Duplicate detection is exact-match based on SPLASH hash — near-duplicates or spectra with minor peak differences will not be flagged57- Requires complete spectral metadata (SMILES, InChI, InChIKey, precursor m/z, adduct); spectra lacking any of these are removed before deduplication occurs, potentially discarding useful spectral data58- FragHub has removed macOS and Linux support as of version 1.4.1, restricting execution to Windows x645960## Evidence6162- [other] FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset.: "FragHub refactors duplicate removal by identifying spectra with identical SPLASH keys, enabling systematic deduplication of the spectral dataset."63- [other] Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values. For each group, retain a single representative spectrum (e.g., first occurrence) and mark duplicates for removal.: "Load the spectral dataset (in FragHub JSON format) using Python 3.12. Compute or retrieve the SPLASH key for each spectrum using the spectra-hash tool. Group spectra by identical SPLASH key values."64- [readme] The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate blocks that define different layers of information, separated by dashes.: "The SPLASH (SPectraL hASH) is an unambiguous, database-independent spectral identifier, just as the InChIKey is designed to serve as a unique identifier for chemical structures. It contains separate"65- [readme] All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub.: "All spectra deemed inconsistent, i.e., those lacking SMILES and InChI, precursor m/z, and adduct information, are removed during the processing by FragHub."66- [other] direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub.: "direct integration of spectra-hash (https://github.com/berlinguyinca/spectra-hash) into fraghub."
HolobiomicsLab 