HolobiomicsLab embedding-vector-similarity-ranking
Summary
Rank spectrum candidates by computing cosine similarity between a query spectrum's Word2vec embedding and a pre-indexed set of reference spectrum embeddings using HNSW approximate nearest-neighbor search. This enables ultra-fast molecular identification from million-scale in-silico libraries while maintaining accuracy.
When to use
You have a query mass spectrum and need to retrieve the most chemically similar reference spectra from a large experimental or predicted library (thousands to millions of entries). The query spectrum has been converted to a Word2vec embedding vector, and reference spectra embeddings are already indexed in an HNSW structure. Use this skill to obtain a ranked list of candidate matches ordered by similarity, enabling rapid molecular identification without exhaustive pairwise comparison.
When NOT to use
- Query spectrum has not yet been embedded using the same Word2vec model as the reference library; embeddings must be generated by the identical pre-trained model to ensure comparability.
- HNSW index has not been built from the reference spectra; you must first construct the index using hnswlib before querying.
- Reference library size is very small (< 1000 spectra); exhaustive nearest-neighbor search may be simpler and equally fast without index overhead.
Inputs
- Query spectrum embedding vector (Word2vec-encoded, floating-point)
- Pre-built HNSW index binary file (e.g., references_index.bin)
- Reference spectrum embeddings database (pre-indexed)
- Number of candidates k to retrieve
Outputs
- Ranked list of nearest-neighbor spectrum identifiers
- Similarity scores (cosine distance) for each retrieved spectrum
- Top-k candidate spectra ordered by chemical similarity
How to apply
Load the pre-built HNSW index from a binary file (e.g., 'references_index.bin') and the query spectrum embedding vector (Word2vec-encoded). Call the HNSW approximate nearest-neighbor search method to retrieve the k most similar candidate spectra, ranked by cosine distance. The HNSW index uses default navigable small world graph parameters (M and ef settings) to balance speed and accuracy during construction and query time. Return the ranked list of spectrum identifiers with their corresponding similarity scores. Cosine distance is the default metric; inspect the returned rankings to confirm high-confidence matches (typically cosine similarity > 0.7 for correct molecular identifications in mass spectrometry).
Related tools
- hnswlib (Implements hierarchical navigable small world graph data structure for approximate nearest-neighbor search on embedding vectors with configurable M and ef parameters) — https://github.com/nmslib/hnswlib
- gensim (Provides Word2vec model training and vector encoding for spectrum embedding generation)
- Python 3.7 (Runtime environment for loading and querying HNSW indexes and spectrum embeddings)
Examples
import hnswlib; index = hnswlib.Index(space='cosine', dim=100); index.load_index('references_index.bin'); labels, distances = index.knn_query(query_embedding, k=10)
Evaluation signals
- Retrieved spectrum identifiers are unique and belong to the indexed reference library (no out-of-bounds or duplicate returns).
- Similarity scores are in valid cosine distance range (0.0 to 1.0 or equivalent normalized metric); scores decrease monotonically across the ranked list.
- Top-ranked spectrum matches correspond to known correct molecular identifications when validated against experimental or curated reference data (e.g., NIST 2017 or MassBank libraries).
- Query runtime is sub-second for k ≤ 100 on million-scale libraries, confirming HNSW acceleration over brute-force search.
- Recall of true matches within top-k is > 95% compared to exhaustive nearest-neighbor search, validating approximate search approximation quality.
Limitations
- HNSW is an approximate nearest-neighbor method; rank order may differ from exhaustive nearest-neighbor search, particularly for lower-ranked candidates.
- Query accuracy depends critically on the quality of the Word2vec embedding model and consistency between query and reference embeddings; mismatched or poorly trained models degrade ranking quality.
- HNSW index is immutable after construction; adding new reference spectra requires rebuilding the entire index, which may be computationally expensive for very large libraries.
- Platform support is currently limited; FastEI software is tested only on Windows 7 and Windows 10 64-bit systems as of the latest release.
Evidence
- [other] For a given query spectrum embedding, perform approximate nearest-neighbor search on the HNSW index to retrieve the k most similar candidate spectra (ranked by cosine distance).: "For a given query spectrum embedding, perform approximate nearest-neighbor search on the HNSW index to retrieve the k most similar candidate spectra (ranked by cosine distance)."
- [other] FastEI implements HNSW-based indexing to enable approximate nearest-neighbor search on Word2vec-embedded spectrum vectors, allowing rapid retrieval from million-scale in-silico libraries while maintaining accuracy.: "FastEI implements HNSW-based indexing to enable approximate nearest-neighbor search on Word2vec-embedded spectrum vectors, allowing rapid retrieval from million-scale in-silico libraries while"
- [other] Initialize an HNSW index using hnswlib with default parameters (M and ef settings for navigable small world graph construction).: "Initialize an HNSW index using hnswlib with default parameters (M and ef settings for navigable small world graph construction)."
- [readme] FastEI is an ultra-fast and accurate spectrum matching method, proposed to improve accuracy by Word2vec-based spectrum embedding and boost the speed using hierarchical navigable small world graph (HNSW): "FastEI is an ultra-fast and accurate spectrum matching method, proposed to improve accuracy by Word2vec-based spectrum embedding and boost the speed using hierarchical navigable small world graph"
- [readme] a million-molecule scale in-silico library has been builded and an ultra-fast and accurate search method has been developed (FastEI): "a million-molecule scale in-silico library has been builded and an ultra-fast and accurate search method has been developed (FastEI)"