Corpus Bag Of Words Representation

corpus-bag-of-words-representation

Summary

Convert LC-MS/MS metabolomics spectra from MGF format into a bag-of-words corpus representation suitable for topic modeling. This serializes each MS2 spectrum as a dictionary of mass-to-charge (m/z) peak features and their abundances, enabling downstream latent Dirichlet allocation (LDA) analysis.

When to use

When you have raw LC-MS/MS data in MGF format and need to prepare it for unsupervised topic modeling of metabolomics fragmentation patterns. Apply this skill as the mandatory first step before running gensim LDA, to extract spectral features (m/z peaks) as a corpus suitable for probabilistic topic inference.

When NOT to use

• Input is already a feature table or pre-computed document-term matrix (skip directly to LDA)
• Data is in a non-MS2 format (e.g., quantification tables, gene expression matrices) that does not represent fragmentation spectra
• You require retention of raw spectrum metadata (e.g., retention time, precursor m/z, scan IDs) for downstream analysis—this step produces only bag-of-words tokens

Inputs

• MGF file (Mascot Generic Format) containing LC-MS/MS spectra with MS2 fragmentation data and m/z peak lists

Outputs

• JSON corpus file containing bag-of-words representation of spectra (list of (feature_index, abundance) pairs per spectrum)
• Feature dictionary mapping m/z peaks to integer indices

How to apply

Load the MGF file using gensim's corpus generation module, then parse MS2 fragmentation spectra to extract mass-to-charge ratio (m/z) peaks as feature tokens. Build a dictionary mapping each unique peak to a feature index. Serialize the resulting bag-of-words representation—where each spectrum is encoded as a list of (feature_index, abundance) pairs—into a JSON corpus file. The corpus JSON format must be compatible with gensim's downstream LDA and serialization modules, preserving the spectral feature vocabulary for consistent topic inference.

Related tools

• gensim (Provides corpus generation module and bag-of-words serialization for MS2 feature extraction and LDA-compatible dictionary building) — https://github.com/sdrogers/ms2ldaviz
• Python (Runtime environment and primary scripting language for corpus generation via run_gensim.py) — https://github.com/sdrogers/ms2ldaviz
• run_gensim.py (Command-line tool that orchestrates MGF parsing, m/z feature extraction, dictionary creation, and JSON corpus serialization) — https://github.com/sdrogers/ms2ldaviz

Examples

./run_gensim.py corpus -f mgf myexp.mgf myexp.corpus.json

Evaluation signals

• Output JSON file is valid and parseable, containing a list of documents (spectra) each represented as (feature_index, count) pairs
• Feature dictionary size and peak coverage reflect the expected m/z range and complexity of the input spectra (e.g., verify against raw MGF peak counts)
• Downstream gensim LDA step runs without vocabulary mismatch errors, confirming corpus feature indices align with dictionary
• Corpus sparsity and token distribution are reasonable (majority of spectra should have multiple peaks, no single spectrum dominates feature space)
• Re-running corpus generation on the same MGF produces identical JSON output (deterministic feature ordering and indexing)

Limitations

• Bag-of-words representation discards peak ordering, intensity magnitude scaling, and spectral metadata (retention time, precursor m/z); may lose signal for methods requiring temporal or structural context.
• Peak discretization via m/z binning into integer indices can lose fine-grained mass accuracy; choice of m/z tolerance or binning strategy is not parameterized in the cited command.
• MGF format parsing assumes well-formed input; malformed spectra (missing peaks, invalid m/z values) may cause silent errors or vocabulary inconsistencies.
• Memory scaling with corpus size is not discussed; very large MGF files may exceed gensim's in-memory dictionary building capacity.

Evidence

• [other] The corpus generation step accepts an MGF-format MS2 file and outputs a JSON corpus representation via the command: ./run_gensim.py corpus -f mgf myexp.mgf myexp.corpus.json: "corpus generation step accepts an MGF-format MS2 file and outputs a JSON corpus representation via the command: ./run_gensim.py corpus -f mgf myexp.mgf myexp.corpus.json"
• [other] Parse MS2 fragmentation spectra and extract mass-to-charge ratio (m/z) peaks as features. Build a dictionary mapping peaks to feature indices. Serialize the corpus as a JSON file containing the bag-of-words representation of each spectrum using the gensim corpus format.: "Parse MS2 fragmentation spectra and extract mass-to-charge ratio (m/z) peaks as features. Build a dictionary mapping peaks to feature indices. Serialize the corpus as a JSON file containing the"
• [readme] Perform 3 steps: 1. Generate corpus/features from MS2 file 2. Run lda using gensim 3. Insert lda result into db: "Performs 3 steps: 1. Generate corpus/features from MS2 file 2. Run lda using gensim 3. Insert lda result into db"
• [intro] visualise how topics inferred from Latent Dirichlet Allocation can be used to assist in the unsupervised characterisation of fragmented (LC-MS-MS) metabolomics data: "visualise how topics inferred from Latent Dirichlet Allocation can be used to assist in the unsupervised characterisation of fragmented (LC-MS-MS) metabolomics data"