HolobiomicsLab hyperdimensional-vector-encoding
Summary
Encodes mass spectra into binary hypervectors of ultra-high dimensionality (>1000) using level-id encoding, enabling lightweight yet powerful data representation for downstream clustering. This brain-inspired approach projects m/z–intensity pairs into hyperdimensional space to improve clustering quality and speed.
When to use
Apply this skill when you have preprocessed mass spectra (mz/intensity pairs in MGF format) and need to convert them into a compact, fixed-dimensional representation suitable for fast similarity computation and clustering. The technique is especially valuable when clustering large-scale datasets (millions of spectra) where traditional distance metrics become computationally expensive.
When NOT to use
- Input spectra are already in feature vector form (e.g., fingerprints, already-embedded representations) — encoding is redundant.
- You require interpretability of individual dimensions or direct correlation to m/z bins — hyperdimensional encoding obscures peak-level attribution.
- GPU memory is unavailable and CPU-only encoding is mandatory — the method is optimized for GPU and slower on CPU alone.
Inputs
- MGF files containing raw mass spectra (mz/intensity peak pairs)
- Preprocessed spectra with filtering applied (peak count, intensity thresholds, m/z range)
- Configuration parameters: HD dimensionality, quantization level, batch size
Outputs
- Binary hypervectors (high-dimensional, typically 2048–8192 dimensions)
- Encoded spectra checkpoint file (optional parquet or binary format)
- Metadata linking each hypervector to original spectrum identifiers and precursor properties
How to apply
Load raw mass spectra from MGF input files containing mz/intensity peak pairs. Apply level-id encoding to project each spectrum into a binary hypervector of configurable dimensionality (default: 2048) with quantization level (default: 16). The encoding is GPU-accelerated and batched for efficiency (default batch size: 5000). Each spectrum is transformed into a high-dimensional binary representation that preserves spectral similarity relationships while enabling Hamming distance computation. Validate that output hypervectors are binary format and match the specified HD dimensionality before export.
Related tools
- HyperSpec (Python library providing GPU-accelerated level-id encoding and HD projection for mass spectra) — https://github.com/wh-xu/Hyper-Spec
- Python 3.8+ (Language runtime for encoding pipeline and CUDA integration)
- CUDA (GPU compute platform enabling parallel hypervector encoding and Hamming distance computation)
Examples
python src/main.py ~/dataset/ ./output.csv --cpu_core_preprocess=4 --hd_dim=2048 --hd_Q=16 --batch_size=5000 --use_gpu_cluster --cluster_alg dbscan
Evaluation signals
- Output hypervectors are binary (all values 0 or 1) and match the configured HD dimensionality exactly.
- Hamming distances between encoded spectra are computable in O(n/64) time per pair (bitwise operations), confirming binary format.
- Spectra with high cosine similarity in original m/z space produce low Hamming distance in HD space (inverse relationship validation).
- Encoding batch throughput meets expected GPU performance (e.g., >5000 spectra/second on GTX 3090) indicating correct GPU utilization.
- Clustering results on encoded vectors show comparable or improved silhouette score and purity compared to baseline clustering on raw spectra.
Limitations
- Encoding quality depends on careful tuning of quantization level (HD_Q) and dimensionality (HD_DIM); suboptimal settings reduce clustering coherence.
- Large-scale datasets (>25M spectra) require high-memory GPU (GTX 3090 recommended; GTX 1080Ti sufficient for smaller datasets per README).
- Brain-inspired level-id encoding is a lossy projection; original m/z values cannot be recovered from hypervectors.
- GPU availability is mandatory for production performance; CPU fallback is orders of magnitude slower.
Evidence
- [readme] HyperSpec first encodes the processed spectra into binary hypervector (HV) with ultra-high dimension (>1000) based on level-id encoding method.: "HyperSpec first encodes the processed spectra into binary hypervector (HV) with ultra-high dimension (>1000) based on level-id encoding method."
- [intro] HyperSpec adopts brain-inspired hyperdimensional computing to project spectra data into binary hyperdimensional space for better clustering quality and faster clustering speed.: "HyperSpec adopts the brain-inspired hyperdimensional (HD) computing to project the spectra data into binary hyperdimensional space to obtain better clustering quality and faster clustering speed."
- [other] Apply brain-inspired HD computing projection to encode each spectrum into a binary hyperdimensional vector representation; validate encoding dimensionality and binary format compliance.: "Apply brain-inspired hyperdimensional (HD) computing projection to encode each spectrum into a binary hyperdimensional vector representation. 3. Validate encoding dimensionality and binary format"
- [readme] The HD dimension defaults to 2048 and quantization level defaults to 16.: "--hd_dim The HD dimension. (default: 2048)"
- [intro] HyperSpec shortens runtime on 25 million spectra from hours to <15 minutes using HD encoding and GPU acceleration.: "HyperSpec shortens the runtime on the draft human proteome dataset with 25 million spectra from a few hours to <15 minutes."
- [readme] The encoding module is implemented and optimized for GPU for shorter runtime.: "The encoding module is implemented and optimized for GPU for shorter runtime."