# Gensim Pipeline Execution > Use when you have an MS2 spectral file (MGF format) from LC-MS/MS metabolomics analysis and need to discover latent topics across fragmentation patterns for unsupervised characterization. Use it as the prerequisite step before topic visualization in the ms2lda web interface. - Skill: `holobiomicslab/gensim-pipeline-execution` (Agent Skill) - Install (CLI): `npx skillmds@latest add holobiomicslab/gensim-pipeline-execution` - Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/gensim-pipeline-execution/raw - Safety review: PASS (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: CC-BY-4.0 - Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/holobiomicslab/gensim-pipeline-execution --- # gensim-pipeline-execution ## Summary Execute a three-stage gensim LDA pipeline to infer topics from LC-MS/MS metabolomics data: generating a corpus from MS2 spectra, running latent Dirichlet allocation, and loading results into PostgreSQL. This skill orchestrates the complete workflow from raw mass spectrometry input to database-backed topic visualization. ## When to use Apply this skill when you have an MS2 spectral file (MGF format) from LC-MS/MS metabolomics analysis and need to discover latent topics across fragmentation patterns for unsupervised characterization. Use it as the prerequisite step before topic visualization in the ms2lda web interface. ## When NOT to use - Input is already a pre-computed LDA model or topic table — use the insert endpoint directly instead. - MS2 file is in a format other than MGF — first convert to MGF or use appropriate preprocessing. - PostgreSQL or Django server is not running — set up Docker containers or local services first. ## Inputs - MS2 spectral file (MGF format) - PostgreSQL server connection (running and accessible) - Django server (up and running, as noted in README) ## Outputs - Corpus JSON file (intermediate: feature representation from MS2 data) - LDA result JSON file (gensim model: topic distributions and model parameters) - PostgreSQL database records (experiment metadata, topic assignments, LDA artifacts) ## How to apply Execute three sequential commands via run_gensim.py: (1) corpus generation from an MGF file with `./run_gensim.py corpus -f mgf `, which extracts features/corpus representation; (2) gensim LDA inference with `./run_gensim.py gensim `, which infers topic distributions; (3) database insertion with `./run_gensim.py insert `, which loads the JSON result into PostgreSQL for web access. Verify success by confirming no errors in each stage and that the experiment is accessible in the database. ## Related tools - **gensim** (Performs Latent Dirichlet Allocation topic inference on the corpus representation of MS2 spectra) - **PostgreSQL** (Stores LDA results, experiment metadata, and topic assignments for persistent access and web visualization) - **Django** (Web application framework hosting the ms2lda server that provides access to inserted LDA experiments) — https://github.com/glasgowcompbio/ms2ldaviz - **Python** (Language in which run_gensim.py and the pipeline orchestration are implemented) - **Docker** (Containerization for PostgreSQL and other services required by the ms2lda application) ## Examples ``` ./run_gensim.py corpus -f mgf myexp.mgf myexp.corpus.json && ./run_gensim.py gensim myexp.corpus.json myexp.ldaresult.json && ./run_gensim.py insert myexp.ldaresult.json stefanv myexp ``` ## Evaluation signals - No error messages or exceptions from any of the three run_gensim.py commands (corpus, gensim, insert). - Corpus JSON file is valid JSON and contains topic term distributions matching the input MGF feature count. - LDA result JSON file is parseable and includes model parameters (alpha, beta, topic distributions). - PostgreSQL query confirms that the experiment record exists with the provided username and experiment name. - Web interface (http://localhost:8000) returns the experiment when queried or browsed post-insertion. ## Limitations - Requires a running Django server and PostgreSQL database — offline or disconnected workflows will fail at the insert stage. - The pipeline is synchronous and blocking; large MGF files may require significant computation time for LDA inference. - Input must be MGF format; other MS2 formats (e.g., mzML) require external conversion before corpus generation. ## Evidence - [intro] topics inferred from Latent Dirichlet Allocation can be used to assist in the unsupervised characterisation of fragmented (LC-MS-MS) metabolomics data: "topics inferred from Latent Dirichlet Allocation can be used to assist in the unsupervised characterisation of fragmented (LC-MS-MS) metabolomics data" - [readme] Performs 3 steps: 1. Generate corpus/features from MS2 file 2. Run lda using gensim 3. Insert lda result into db: "Performs 3 steps: 1. Generate corpus/features from MS2 file 2. Run lda using gensim 3. Insert lda result into db" - [readme] ./run_gensim.py corpus -f mgf myexp.mgf myexp.corpus.json ./run_gensim.py gensim myexp.corpus.json myexp.ldaresult.json ./run_gensim.py insert myexp.ldaresult.json stefanv myexp: "./run_gensim.py corpus -f mgf myexp.mgf myexp.corpus.json ./run_gensim.py gensim myexp.corpus.json myexp.ldaresult.json ./run_gensim.py insert myexp.ldaresult.json stefanv myexp" - [readme] Requires server to be up and running.: "Requires server to be up and running." - [other] The insert step consumes the LDA result JSON file (myexp.ldaresult.json) and loads it into the database using the command ./run_gensim.py insert with parameters for the result file, a username (stefanv), and an experiment name (myexp).: "The insert step consumes the LDA result JSON file (myexp.ldaresult.json) and loads it into the database using the command ./run_gensim.py insert with parameters for the result file, a username"