# Duplicate Spectrum Detection Removal

> Use when you have a large collection of MS/MS spectra with harmonized metadata and suspect that spectra for the same compound or adduct acquired under similar conditions may be duplicated or near-identical.

- Skill: `holobiomicslab/duplicate-spectrum-detection-removal` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/duplicate-spectrum-detection-removal`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/duplicate-spectrum-detection-removal/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/duplicate-spectrum-detection-removal

---


# duplicate-spectrum-detection-removal

## Summary

Identify and remove redundant MS/MS spectra from a harmonized spectral library by calculating pairwise cosine similarity scores and filtering based on a coherence threshold. This skill ensures the final library contains only unique, high-quality spectra for metabolomics applications.

## When to use

Apply this skill when you have a large collection of MS/MS spectra with harmonized metadata and suspect that spectra for the same compound or adduct acquired under similar conditions may be duplicated or near-identical. This typically occurs after metadata standardization and before final library export, as a quality control step to remove redundancy that inflates library size without improving annotation coverage.

## When NOT to use

- Input spectra lack standardized metadata fields (compound ID, adduct); duplicate detection relies on consistent annotations and will produce false positives or false negatives if metadata is incomplete or non-standard.
- You need to retain all raw spectra for independent validation or method comparison studies; deduplication is irreversible and removes records that might be scientifically valuable despite redundancy.
- Cosine similarity threshold has not been validated for your specific instrument platform or ionization mode; applying a generic threshold without prior calibration may over-deduplicate or under-deduplicate.

## Inputs

- MGF files with standardized metadata (compound identifier, adduct annotation)
- CSV files containing spectrum-level metadata and quality metrics

## Outputs

- Deduplicated MGF file with candidate duplicates removed
- Numpy arrays of pairwise cosine similarity scores
- Log file documenting which spectra were retained and which were merged or discarded

## How to apply

First, identify candidate duplicate spectra by grouping on compound identifiers and adduct annotations using step 3-3 (uniq-comb.py). Second, calculate pairwise cosine similarity scores for all candidate pairs and store results in numpy format (step 3-4, uniq-cos-calc.py), using the SpectralEntropy library's cosine score implementation. Third, apply a cosine similarity threshold—spectra with scores above this threshold are marked as duplicates—and retain only the highest-quality spectrum from each duplicate group based on user-defined criteria such as spectral completeness or signal-to-noise ratio (step 3-5, uniq-select.py). The rationale is that spectra sharing identical or near-identical fragment patterns and metadata indicate redundant measurements, and keeping only one representative per group reduces library bloat while preserving annotation diversity.

## Related tools

- **matchms** (Preprocessing and scoring of MS/MS spectra; used before and after deduplication to repair and validate metadata coherence) — https://github.com/matchms/matchms
- **SpectralEntropy** (Calculation of cosine similarity scores between pairs of spectra to quantify redundancy) — https://github.com/YuanyueLi/SpectralEntropy.git
- **spectraverse-analysis** (Repository containing the complete deduplication pipeline (steps 3-3 through 3-5) and configuration) — https://github.com/skinniderlab/spectraverse-analysis

## Examples

```
python run_steps.py --config config/config_step3.json
```

## Evaluation signals

- Pairwise cosine similarity matrix is square and symmetric with diagonal values equal to 1.0, confirming that spectra are only compared against other spectra, not against themselves
- No spectra are retained from a duplicate group if all members score below the cosine threshold; all spectra in a group above the threshold belong to the same deduplicated record
- The output MGF file has fewer total spectra than the input, and the reduction count matches the number of spectra flagged as duplicates in the log file
- Compound identifiers and adduct annotations remain unchanged in the deduplicated output; only redundant spectra are removed, not metadata fields
- Visual inspection of a sample of retained spectra shows visually distinct fragmentation patterns, while removed spectra show near-identical peak lists and intensities to their retained counterparts

## Limitations

- Cosine similarity is sensitive to mass accuracy and intensity normalization; spectra with poor mass calibration or non-standard intensity scaling may be incorrectly classified as duplicates or singletons
- Deduplication is performed only within compound–adduct groups; spectra of the same compound in different adduct states (e.g., [M+H]+ vs. [M+Na]+) are not compared and will not be deduplicated, which is correct for annotation purposes but may not reduce library redundancy if the same ion was measured in multiple adduct forms
- No changelog is available in the repository, making it difficult to track changes to the cosine similarity calculation or deduplication thresholds across versions
- The threshold for cosine similarity must be manually set; the README provides no guidance on calibration, and an inappropriate threshold can lead to over-aggressive or insufficient deduplication

## Evidence

- [readme] Identification of candidate duplicate spectra: "step3-3_uniq-comb.py: Identification of candidate duplicate spectra"
- [readme] Cosine score calculation for deduplication: "step3-4_uniq-cos-calc.py: Calculating pairwise cosine scores for duplicate spectra and saving in numpy format"
- [readme] Removal of duplicates based on cosine threshold: "step3-5_uniq-select.py: Removal of duplicate spectra based on pairwise cosine scores"
- [readme] SpectralEntropy for cosine calculation: "# Code used to calculate cosine score
   git clone https://github.com/YuanyueLi/SpectralEntropy.git"
- [intro] Purpose of deduplication in pipeline: "code used to preprocess and harmonize MS/MS spectra and their associated metadata in the compilation of Spectraverse"