# Proteome Dataset Handling

> Use when you have a collection of MS/MS spectra in MGF format and need to prepare them for GPU-based clustering. Dataset size and available GPU memory are critical: use GTX 1080Ti for smaller proteome datasets; use GTX 3090 for datasets like PXD000561 that exceed GTX 1080Ti capacity.

- Skill: `holobiomicslab/proteome-dataset-handling` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/proteome-dataset-handling`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/proteome-dataset-handling/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/proteome-dataset-handling

---


# proteome-dataset-handling

## Summary

Configure and load mass spectrometry proteome datasets (MGF format) into a GPU-accelerated clustering pipeline, matching dataset scale to available GPU memory (GTX 1080Ti for smaller datasets, GTX 3090 for large-scale datasets like PXD000561 with millions of spectra).

## When to use

You have a collection of MS/MS spectra in MGF format and need to prepare them for GPU-based clustering. Dataset size and available GPU memory are critical: use GTX 1080Ti for smaller proteome datasets; use GTX 3090 for datasets like PXD000561 that exceed GTX 1080Ti capacity. Preprocessing must normalize spectra by removing precursor peaks, filtering by intensity thresholds, and binning m/z values before encoding into hypervectors.

## When NOT to use

- Input spectra are already in hyperdimensional vector format or have been encoded by another method
- Dataset is smaller than a few thousand spectra and does not require GPU acceleration
- Spectra are in formats other than MGF (mzML, mzXML, or other formats must be converted first)

## Inputs

- Directory containing MGF files (mass spectrometry peak lists)
- ProteomeXchange dataset identifiers (e.g., PXD000561)
- Preprocessing parameter configuration (peak filtering, intensity scaling, tolerance values)

## Outputs

- Preprocessed spectra checkpoint file (HyperSpec checkpoint format)
- Encoded hypervectors in hyperdimensional space (binary representations)
- Dataset metadata: precursor m/z, charge state, retention time per spectrum

## How to apply

First, verify your GPU type matches the dataset scale (GTX 3090 for large datasets like PXD000561, GTX 1080Ti for smaller proteomes). Second, organize all MGF files in a single input directory. Third, execute the HyperSpec preprocessing pipeline with appropriate CPU core allocation (default: 6 cores) and filtering parameters: set --min_peaks to exclude low-quality spectra, --min_intensity to filter noise, --max_peaks_used to cap high-intensity peaks, and --remove_precursor_tol (typically 0.5 Da) to eliminate precursor contamination. Fourth, apply intensity scaling (root, log, rank, or off) based on your spectral characteristics. Finally, configure charge filtering (--cluster_charges 2 3) to focus on the most abundant charge states. The preprocessing stage projects spectra into normalized feature space before GPU-accelerated hyperdimensional encoding.

## Related tools

- **HyperSpec** (GPU-accelerated mass spectra clustering framework that loads and preprocesses MGF files into hyperdimensional space) — https://github.com/wh-xu/Hyper-Spec
- **Python** (Runtime environment for HyperSpec execution; version 3.8+ required)
- **CUDA** (GPU compute framework enabling GPU-accelerated preprocessing and clustering)

## Examples

```
python src/main.py ~/dataset/ ./output.csv --cpu_core_preprocess=4 --min_peaks=10 --min_intensity=1.0 --max_peaks_used=100 --remove_precursor_tol=0.5 --scaling=root --cluster_charges 2 3
```

## Evaluation signals

- Input MGF directory is accessible and contains valid peak list files with m/z and intensity pairs
- Preprocessing pipeline completes without errors; checkpoint file is created or batch encoding proceeds
- Output spectra metadata includes valid ranges: precursor_mz > 0, precursor_charge ∈ {1,2,3,...}, retention_time ≥ 0
- Number of spectra after filtering matches expected retention (e.g., after charge filtering and peak count thresholds)
- GPU memory utilization remains within limits for the selected GPU type (no out-of-memory errors during encoding)

## Limitations

- Clustering for large datasets like PXD000561 requires GTX 3090 GPU with larger memory; GTX 1080Ti will fail on such datasets
- System requires Linux platform with properly configured CUDA environment; Windows and macOS support not documented
- HyperSpec has been tested only on GTX 1080Ti and GTX 3090; other NVIDIA GPUs may require additional validation
- High-performance SSD storage is recommended for best performance; standard storage may create I/O bottlenecks with large datasets
- MGF format is required; other mass spectrometry formats (mzML, mzXML) must be converted beforehand

## Evidence

- [readme] System Requirements: _HyperSpec_ requires `Python 3.8+` with `CUDA` environment: "_HyperSpec_ requires `Python 3.8+` with `CUDA` environment. A GPU should be installed properly."
- [readme] GPU allocation by dataset scale: "Clustering for PXD000561 dataset requires GTX 3090 with larger memory. Clustering for other dataset with smaller scale requires GTX 1080Ti."
- [readme] Input format and preprocessing workflow: "_HyperSpec_ supports running using the command line and takes `MGF` peak files as input and exports the clustering result as a CSV file"
- [readme] Preprocessing parameters for dataset handling: "--cpu_core_preprocess [CPU_CORE_PREPROCESS], --min_peaks [MIN_PEAKS], --min_intensity [MIN_INTENSITY], --max_peaks_used [MAX_PEAKS_USED], --scaling {off,root,log,rank}, --remove_precursor_tol"
- [readme] Performance-critical infrastructure recommendation: "We recommend using high-performance SSD as the storage device for the best performance."
- [readme] Brain-inspired hyperdimensional encoding and bucketing: "_HyperSpec_ first encodes the processed spectra into binary hypervector (HV) with ultra-high dimension (>1000) based on level-id encoding method."