feature-vector-similarity-search-preparation
Summary
Convert high-resolution MS/MS spectra to low-dimensional hashed vectors and construct nearest neighbor indexes to enable efficient similarity searching without exhaustive pairwise comparisons. This skill bridges spectrum preprocessing and fast clustering by creating the index structures that make large-scale spectrum analysis computationally feasible.
When to use
Apply this skill when you have millions of MS/MS spectra in mzML, mzXML, or MGF format and need to identify similar spectra for clustering, but exhaustive pairwise cosine-similarity computation would be prohibitively slow. Use it as the intermediate step between spectrum preprocessing (binning, scaling) and density-based clustering.
When NOT to use
- Spectra are already represented as precomputed feature vectors or distance matrices—skip directly to clustering.
- The dataset contains fewer than ~10,000 spectra—exhaustive pairwise comparison may be faster than index construction overhead.
- Spectrum format is not mzML, mzXML, or MGF; conversion to these formats is required first.
- You require exact (not approximate) cosine similarities; feature hashing introduces approximation error that depends on
low_dim setting.
Inputs
- High-resolution MS/MS spectra (mzML, mzXML, or MGF format)
- Precursor m/z values for each spectrum
- Preprocessed spectrum intensities (scaled if applicable)
Outputs
- Low-dimensional hashed spectrum vectors (length controlled by
low_dim parameter)
- Serialized nearest neighbor index structures (one per precursor m/z bucket)
- Inverted index mappings for fast neighbor retrieval
How to apply
First, bin high-resolution spectra into sparse vectors capturing fragment mass bins, then apply feature hashing (MurmurHash3) to compress these sparse, high-dimensional vectors into low-dimensional approximations that preserve cosine similarity. Partition hashed vectors into buckets stratified by precursor m/z to create separate nearest neighbor indexes for each bucket, using Voronoi partitioning to assign vectors to nearest representatives in an inverted index structure. Tune the low_dim parameter (length of hashed vectors) to balance approximation accuracy against memory and search speed; larger values more accurately preserve cosine distance but increase computational cost. Tune n_neighbors and n_neighbors_ann (final and ANN-search neighbor counts) to balance clustering completeness against search time. The resulting indexes enable sparse pairwise distance matrix computation by querying only promising neighbors rather than all spectra.
Related tools
- falcon (Command-line spectrum clustering tool that orchestrates feature hashing, nearest neighbor indexing, and density-based clustering; implements the full workflow from raw spectra to cluster assignments.) — https://github.com/bittremieux/falcon
- spectrum-utils (Python library for spectrum preprocessing (binning, scaling, filtering) prior to hashing and indexing.)
Examples
falcon peak/*.mzml falcon --export_representatives --precursor_tol 20 ppm --fragment_tol 0.05 --eps 0.10
Evaluation signals
- Index construction completes without memory errors and produces serialized index files for each precursor m/z bucket.
- Hashed vectors preserve relative cosine similarity: cosine(hashed_spec_A, hashed_spec_B) ≈ cosine(original_spec_A, original_spec_B) with error bounded by
low_dim choice.
- Nearest neighbor queries retrieve true nearest neighbors within the setting of
n_probe (number of inverted index lists inspected); verify by spot-checking against brute-force k-NN on a subset.
- Downstream density-based clustering (DBSCAN with
eps threshold) produces clusters with >90% purity (clusters contain spectra from single peptide/compound) and >80% sensitivity (true co-fragmenting spectra are grouped together).
- Wall-clock time for clustering millions of spectra is at least 5–10× faster than would be required for exhaustive pairwise cosine computation.
Limitations
- Feature hashing introduces approximation error in cosine similarity; the magnitude depends on
low_dim: smaller dimensions are faster but less accurate. Values that work well for one dataset's spectral characteristics may require tuning for metabolomics vs. proteomics data.
- Inverted index tuning via
n_probe trades accuracy for speed; exploring fewer index lists during neighbor retrieval increases the risk of missing true nearest neighbors in high-dimensional space, potentially leading to fragmented clusters.
- Index construction must stratify by precursor m/z to avoid comparing spectra with incompatible precursor masses; results are sensitive to
precursor_tol setting (default 20 ppm or Dalton).
- The method assumes spectra have been preprocessed to remove low-quality peaks (e.g.,
min_peaks ≥5, min_mz_range ≥250 m/z); raw, unfiltered spectra can produce poor index quality.
Evidence
- [readme] High-resolution spectra binned and converted to low-dimensional vectors using feature hashing: "First, high-resolution spectra are binned and converted to low-dimensional vectors using feature hashing."
- [readme] Spectrum vectors construct nearest neighbor indexes for fast similarity searching: "Next, the spectrum vectors are used to construct nearest neighbor indexes for fast similarity searching."
- [readme] Nearest neighbor indexes enable sparse pairwise distance matrix without exhaustive comparisons: "The nearest neighbor indexes are used to efficiently compute a sparse pairwise distance matrix without having to exhaustively compare all spectra to each other."
- [readme] Feature hashing uses MurmurHash3 and Voronoi partitioning in inverted index: "the sparse, high-dimensional, vectors are hashed to lower-dimensional vectors by using a hash function (the non-cryptographic MurmurHash3 algorithm) to map the mass bins separately to a small number"
- [readme] Precursor m/z buckets and tunable neighbor parameters control index accuracy/speed tradeoff: "The spectrum vectors in each bucket are partitioned into data subspaces to create a Voronoi diagram, and all vectors are assigned to their nearest representative vector in an inverted index. The"
- [readme] n_probe, n_neighbors, and low_dim are tunable parameters for nearest neighbor indexing: "n_probe: The maximum number of lists in the inverted index to inspect during querying. Inspecting fewer lists will run faster but will give slightly less accurate clustering results. n_neighbors and"
1---2name: feature-vector-similarity-search-preparation3description: Use when you have millions of MS/MS spectra in mzML, mzXML, or MGF format and need to identify similar spectra for clustering, but exhaustive pairwise cosine-similarity computation would be prohibitively slow.4license: CC-BY-4.05---67# feature-vector-similarity-search-preparation89## Summary1011Convert high-resolution MS/MS spectra to low-dimensional hashed vectors and construct nearest neighbor indexes to enable efficient similarity searching without exhaustive pairwise comparisons. This skill bridges spectrum preprocessing and fast clustering by creating the index structures that make large-scale spectrum analysis computationally feasible.1213## When to use1415Apply this skill when you have millions of MS/MS spectra in mzML, mzXML, or MGF format and need to identify similar spectra for clustering, but exhaustive pairwise cosine-similarity computation would be prohibitively slow. Use it as the intermediate step between spectrum preprocessing (binning, scaling) and density-based clustering.1617## When NOT to use1819- Spectra are already represented as precomputed feature vectors or distance matrices—skip directly to clustering.20- The dataset contains fewer than ~10,000 spectra—exhaustive pairwise comparison may be faster than index construction overhead.21- Spectrum format is not mzML, mzXML, or MGF; conversion to these formats is required first.22- You require exact (not approximate) cosine similarities; feature hashing introduces approximation error that depends on `low_dim` setting.2324## Inputs2526- High-resolution MS/MS spectra (mzML, mzXML, or MGF format)27- Precursor m/z values for each spectrum28- Preprocessed spectrum intensities (scaled if applicable)2930## Outputs3132- Low-dimensional hashed spectrum vectors (length controlled by `low_dim` parameter)33- Serialized nearest neighbor index structures (one per precursor m/z bucket)34- Inverted index mappings for fast neighbor retrieval3536## How to apply3738First, bin high-resolution spectra into sparse vectors capturing fragment mass bins, then apply feature hashing (MurmurHash3) to compress these sparse, high-dimensional vectors into low-dimensional approximations that preserve cosine similarity. Partition hashed vectors into buckets stratified by precursor m/z to create separate nearest neighbor indexes for each bucket, using Voronoi partitioning to assign vectors to nearest representatives in an inverted index structure. Tune the `low_dim` parameter (length of hashed vectors) to balance approximation accuracy against memory and search speed; larger values more accurately preserve cosine distance but increase computational cost. Tune `n_neighbors` and `n_neighbors_ann` (final and ANN-search neighbor counts) to balance clustering completeness against search time. The resulting indexes enable sparse pairwise distance matrix computation by querying only promising neighbors rather than all spectra.3940## Related tools4142- **falcon** (Command-line spectrum clustering tool that orchestrates feature hashing, nearest neighbor indexing, and density-based clustering; implements the full workflow from raw spectra to cluster assignments.) — https://github.com/bittremieux/falcon43- **spectrum-utils** (Python library for spectrum preprocessing (binning, scaling, filtering) prior to hashing and indexing.)4445## Examples4647```48falcon peak/*.mzml falcon --export_representatives --precursor_tol 20 ppm --fragment_tol 0.05 --eps 0.1049```5051## Evaluation signals5253- Index construction completes without memory errors and produces serialized index files for each precursor m/z bucket.54- Hashed vectors preserve relative cosine similarity: cosine(hashed_spec_A, hashed_spec_B) ≈ cosine(original_spec_A, original_spec_B) with error bounded by `low_dim` choice.55- Nearest neighbor queries retrieve true nearest neighbors within the setting of `n_probe` (number of inverted index lists inspected); verify by spot-checking against brute-force k-NN on a subset.56- Downstream density-based clustering (DBSCAN with `eps` threshold) produces clusters with >90% purity (clusters contain spectra from single peptide/compound) and >80% sensitivity (true co-fragmenting spectra are grouped together).57- Wall-clock time for clustering millions of spectra is at least 5–10× faster than would be required for exhaustive pairwise cosine computation.5859## Limitations6061- Feature hashing introduces approximation error in cosine similarity; the magnitude depends on `low_dim`: smaller dimensions are faster but less accurate. Values that work well for one dataset's spectral characteristics may require tuning for metabolomics vs. proteomics data.62- Inverted index tuning via `n_probe` trades accuracy for speed; exploring fewer index lists during neighbor retrieval increases the risk of missing true nearest neighbors in high-dimensional space, potentially leading to fragmented clusters.63- Index construction must stratify by precursor m/z to avoid comparing spectra with incompatible precursor masses; results are sensitive to `precursor_tol` setting (default 20 ppm or Dalton).64- The method assumes spectra have been preprocessed to remove low-quality peaks (e.g., `min_peaks` ≥5, `min_mz_range` ≥250 m/z); raw, unfiltered spectra can produce poor index quality.6566## Evidence6768- [readme] High-resolution spectra binned and converted to low-dimensional vectors using feature hashing: "First, high-resolution spectra are binned and converted to low-dimensional vectors using feature hashing."69- [readme] Spectrum vectors construct nearest neighbor indexes for fast similarity searching: "Next, the spectrum vectors are used to construct nearest neighbor indexes for fast similarity searching."70- [readme] Nearest neighbor indexes enable sparse pairwise distance matrix without exhaustive comparisons: "The nearest neighbor indexes are used to efficiently compute a sparse pairwise distance matrix without having to exhaustively compare all spectra to each other."71- [readme] Feature hashing uses MurmurHash3 and Voronoi partitioning in inverted index: "the sparse, high-dimensional, vectors are hashed to lower-dimensional vectors by using a hash function (the non-cryptographic MurmurHash3 algorithm) to map the mass bins separately to a small number"72- [readme] Precursor m/z buckets and tunable neighbor parameters control index accuracy/speed tradeoff: "The spectrum vectors in each bucket are partitioned into data subspaces to create a Voronoi diagram, and all vectors are assigned to their nearest representative vector in an inverted index. The"73- [readme] n_probe, n_neighbors, and low_dim are tunable parameters for nearest neighbor indexing: "n_probe: The maximum number of lists in the inverted index to inspect during querying. Inspecting fewer lists will run faster but will give slightly less accurate clustering results. n_neighbors and"
