Baseline Model Implementation For Comparison

baseline-model-implementation-for-comparison

Summary

Implement reference baseline models (e.g., NEIMS with FFN and GNN encoders) under strictly equivalent settings—same covariates, hyperparameter sweeps, train/test splits—to enable fair quantitative comparison with novel spectrum prediction methods. This skill ensures reproducible benchmarking and isolates model architectural differences from confounding experimental factors.

When to use

You are introducing a novel spectrum prediction model and need to demonstrate that performance improvements come from architectural innovation rather than experimental advantage. Baseline implementation is required when comparing across NEIMS, MassFormer, 3DMolMS, GrAFF-MS, or CFM-ID on the same dataset (e.g., NIST'20) with tandem mass spectrum prediction as the evaluation task.

When NOT to use

• Input dataset or splits differ from the novel model's (comparison requires identical data provenance and preprocessing).
• Hyperparameter sweeps for the baseline are not exhaustively identical to the novel model (unfair advantage to one method).
• Baseline source code or paper is unavailable or ambiguously specified (use only published methods with reproducible details).

Inputs

• Molecular structures as SMILES strings or chemical graphs (PubChem records)
• Experimental tandem mass spectra in standardized format (.SDF, .mgf, .hdf5)
• Dataset split specification (train/validation/test indices with seed)
• Hyperparameter sweep configuration (learning rate, batch size, encoder hidden dims, num_layers)

Outputs

• Trained baseline model checkpoint (.pt or equivalent)
• Spectrum predictions on test set (m/z, intensity tuples or ranking scores)
• Evaluation metrics (top-1/top-k retrieval accuracy, spectral similarity, cosine distance)
• Metadata log (hyperparameters, random seed, training time, hardware used)

How to apply

Select a reference baseline architecture from the literature (e.g., NEIMS FFN or GNN encoder from Goldman et al. 2009). Load the molecular dataset and chemical structures from PubChem or a commercial source (NIST'20) with associated experimental mass spectra in standardized format (.SDF or .hdf5). Construct the encoder (FFN or GNN) and train end-to-end on the spectrum prediction task using the exact same dataset splits, hyperparameter sweep grid, learning rate schedule, and batch settings as your novel model. Evaluate on the held-out test set computing prediction accuracy and comparison metrics (e.g., top-1/top-k retrieval accuracy for structural elucidation, spectral similarity cosine scores). Save model weights, predictions, and evaluation metrics to output artifacts. Document all hyperparameters and random seeds (e.g., split_1_rnd1) to enable replication.

Related tools

• PubChem (Source of molecular structures and chemical formula lookups for dataset construction and retrieval experiments)
• ms-pred (coleygroup) (Reference implementation of NEIMS and other baseline models under equivalent-settings framework for spectrum prediction) — https://github.com/coleygroup/ms-pred
• MAGMa (Annotation tool for generating fragmentation trees and substructure labels used in baseline and novel model training pipelines)

Evaluation signals

• Hyperparameter sweep configuration is identical between baseline and novel model (same ranges, grid resolution, learning rate schedules).
• Train/validation/test split indices and random seeds are reproducible and identical across both implementations.
• Baseline model achieves published or expected performance on the reference test set (e.g., NIST'20 top-1 accuracy ~30–40% for NEIMS GNN).
• Evaluation metrics (top-k retrieval accuracy, cosine similarity scores) are computed on the same held-out test set using identical metric definitions.
• Model checkpoint, predictions, and metadata are saved and can be reloaded to reproduce results without retraining.

Limitations

• NIST'20 is a commercial dataset requiring purchase; open-source alternatives (MassSpecGym) have less manual curation and yield different results.
• Baseline code availability and clarity vary; some published methods (e.g., CFM-ID 4.0) are not retrained in the comparison—only reference results are provided.
• GPU memory and training time scale with dataset size and hyperparameter sweep breadth; smaller GPUs (<24 GB) require reduced batch sizes and may not replicate published results.
• Contrastive finetuning steps (e.g., for ICEBERG intensity models) are optional and affect final performance; equivalent-settings comparison requires explicit choices about which optional steps to include.

Evidence

• [readme] In order to fairly compare various spectra models, we implement a number of baselines and alternative models using equivalent settings across models (i.e., same covariates, hyperparameter sweeps for each, etc.): "In order to fairly compare various spectra models, we implement a number of baselines and alternative models using equivalent settings across models (i.e., same covariates, hyperparameter sweeps for"
• [other] NEIMS baseline is implemented with both FFN and GNN encoder variants as part of an equivalent-settings comparison framework where all models use the same covariates and hyperparameter sweeps.: "NEIMS baseline is implemented with both FFN and GNN encoder variants as part of an equivalent-settings comparison framework where all models use the same covariates and hyperparameter sweeps."
• [other] Construct a graph neural network (GNN) encoder to learn molecular representations from chemical graphs. 3. Train the GNN encoder end-to-end on the spectrum prediction task using the dataset split and hyperparameters specified for the baseline comparison. 4. Evaluate the trained GNN-encoder NEIMS model on the held-out test set, computing prediction accuracy and comparison metrics against ms-pred reference results.: "Construct a graph neural network (GNN) encoder to learn molecular representations from chemical graphs. 3. Train the GNN encoder end-to-end on the spectrum prediction task using the dataset split and"
• [other] Load the molecular dataset and chemical structures from PubChem with associated experimental mass spectra.: "Load the molecular dataset and chemical structures from PubChem with associated experimental mass spectra."
• [readme] 1. NEIMS using both FFN and GNN encoders from Rapid prediction of electron–ionization mass spectrometry using neural networks: "1. NEIMS using both FFN and GNN encoders from Rapid prediction of electron–ionization mass spectrometry using neural networks"