Benchmarking and Comparative Performance Evaluation
Summary
Systematically compare a novel method (MIST-CF) against established baseline approaches (SIRIUS, FFN, Xformer) on standardized benchmark datasets using ranking accuracy metrics (top-1, top-3, top-k). This skill enables quantitative assessment of whether a new chemical formula inference model outperforms existing tools on retrospective and prospective MS/MS data.
When to use
Apply this skill when you have trained a new machine learning model for chemical formula or adduct assignment from MS/MS spectra and need to assess whether it offers genuine performance gains over established baselines. Use it specifically when you have access to ground-truth annotations (e.g., NPLIB1, NIST20, CASMI challenge datasets) and must justify publication or deployment claims with comparative accuracy metrics.
When NOT to use
- Input is a single spectrum or small set without ground-truth annotations — use case-by-case evaluation instead.
- Baseline tool (e.g., SIRIUS) is not available or cannot be run on your hardware/OS — benchmarking requires all competing methods executable in the same environment.
- You are comparing only internal model variants (e.g., ablation study) without a published external baseline — use internal ablation evaluation rather than full benchmarking.
Inputs
- MS/MS spectra in MGF format with precursor m/z values
- Ground-truth chemical formula and adduct type annotations
- Candidate formula lists (generated by SIRIUS decomp or chemical enumeration)
- Trained neural network weights for MIST-CF, FFN, Xformer baselines
- Preprocessed spectral data (normalized intensities, noise-filtered peaks)
Outputs
- Per-spectrum ranking predictions (ranked list of candidate formulas with adduct types)
- Top-1, top-3, top-k accuracy metrics for formula assignment
- Top-1, top-3, top-k accuracy metrics for adduct assignment
- Performance summary table (model × accuracy metric)
- Ranking accuracy plots (e.g., cumulative accuracy vs. rank position)
- Comparative analysis report (MIST-CF vs. SIRIUS, FFN, Xformer)
How to apply
Run inference predictions from all competing models (MIST-CF, SIRIUS decomp, FFN, Xformer) on the same test dataset, ensuring identical MS/MS preprocessing (intensity normalization, noise filtering, precursor m/z extraction). For each spectrum, compute ranked predictions for chemical formula and adduct type; evaluate using ranking metrics (top-1 accuracy: first ranked candidate matches ground truth; top-3, top-k: ground truth appears in top k predictions). Tabulate accuracy results side-by-side and generate ranking accuracy plots showing cumulative performance across the test set. On proprietary datasets (e.g., NIST20), train on disjoint splits; on public benchmarks (NPLIB1, CASMI22), use published train-test boundaries. Document hyperparameter tuning (via hyperopt or grid search) for all models on validation data to ensure fair comparison.
Related tools
- MIST-CF (Novel energy-based neural network model for ranking chemical formulas and adduct types from MS/MS spectra; main method under evaluation.) — https://github.com/samgoldman97/mist-cf
- SIRIUS (Baseline tool for formula enumeration via dynamic programming algorithm (SIRIUS decomp); generates candidate formula lists and provides ranking predictions for comparison.) — https://bio.informatik.uni-jena.de/software/sirius/
- FFN (Feed-forward neural network baseline model for formula ranking; included in comparative benchmarking experiments.)
- Xformer (Transformer baseline model for formula ranking; included in comparative benchmarking experiments.)
- SCARF (Related work on sinusoidal formula embeddings; used as feature representation in MIST-CF architecture.) — https://arxiv.org/abs/2303.06470
Examples
. run_scripts/benchmarking/train_mist_cf.sh && . run_scripts/benchmarking/train_ffn.sh && . run_scripts/benchmarking/train_xformer.sh && python run_scripts/benchmarking/eval_models.py
Evaluation signals
- Top-k accuracy values are monotonically non-decreasing as k increases (top-1 ≤ top-3 ≤ top-5, etc.).
- MIST-CF top-k accuracy meets or exceeds SIRIUS baseline on retrospective (NPLIB1, NIST20) and prospective (CASMI22) test sets, demonstrating improvement over fragmentation-tree-based method.
- Performance metrics are computed on a consistent test split with no data leakage between training and evaluation sets; hyperparameter tuning used only validation data.
- Adduct assignment accuracy is reported separately and is non-trivial (>baseline random selection of adduct types).
- Comparison tables show that all models were tuned to convergence (no model has obviously suboptimal hyperparameters relative to published results or validation curves).
Limitations
- MIST-CF trained on public NPLIB1 dataset may be less performant than NIST20-trained model (particularly for Orbitrap or higher-resolution MS/MS data), limiting generalizability claims.
- Benchmarking focuses on positive-mode MS/MS only; adduct types are still limited beyond [M+H]+, so evaluation does not cover negative-mode or diverse adduct scenarios.
- SIRIUS baseline still required for initial formula enumeration (candidate generation); MIST-CF learns ranking but does not replace formula discovery, limiting end-to-end de novo capability.
- Prospective evaluation (CASMI22) is a single challenge dataset; retrospective benchmarking on NPLIB1 may overfit to that data distribution.
Evidence
- [other] How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS data?: "research_question: How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS"
- [other] Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions.: "4. Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions."
- [other] Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset.: "5. Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset."
- [intro] MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases: "MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases"
- [intro] Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion: "Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion"
- [readme] Retrospective benchmarking: Benchmark MIST-CF performance with baseline models.: "Retrospective benchmarking
Benchmark MIST-CF performance with baseline models."
- [readme] Prospective analysis on CASMI-2022: Compare MIST-CF and SIRIUS on CASMI-2022.: "Prospective analysis: CASMI 2022
Compare MIST-CF and SIRIUS on CASMI-2022."
- [readme] This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data).: "This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data)."
- [intro] Considering multiple adduct types beyond [M+H]+ (still only positive mode): "Considering multiple adduct types beyond [M+H]+ (still only positive mode)"
1---2name: benchmarking-and-comparative-performance-evaluation-33description: Use when you have trained a new machine learning model for chemical formula4 or adduct assignment from MS/MS spectra and need to assess whether it offers genuine5 performance gains over established baselines. Use it specifically when you have6 access to ground-truth annotations (e.7license: CC-BY-4.08metadata:9 source: https://skillmd.com/skills/holobiomicslab/benchmarking-and-comparative-performance-evaluation-310 edam_operation: http://edamontology.org/operation_343611 edam_topics:12 - http://edamontology.org/topic_337213 - http://edamontology.org/topic_012114 - http://edamontology.org/topic_305315 tools:16 - MIST17 - MIST-CF18 - SIRIUS19 - SCARF20 - FFN21 - Xformer22 techniques:23 - LC-MS24 license_tier: open25 provenance_tier: literature26derived_from:27- doi: 10.1021/acs.jcim.3c0108228 title: mistcf29evidence_spans:30- an extension of MIST for annotating MS1 precursor masses from MS/MS data31- MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum32 using an end-to-end energy based modeling approach33- Utilizing an internal chemical subformula assignment protocol (rather than SIRIUS34 fragmentation trees)35- Utilizing sinusoidal formula embeddings as developed in our previous work SCARF36claims: []37provenance:38 collection: https://w3id.org/holobiomicslab/asb-skill/collection/metabolomics/v239 assembled_by: scripts/collect_metabolomics_collection.py40 sources:41 - build: coll_mistcf42 doi: 10.1021/acs.jcim.3c0108243 title: mistcf44 dedup_kept_from: coll_mistcf45schema_version: 0.2.046attribution:47 generator: AgenticScienceBuilder48 original_doi: 10.1021/acs.jcim.3c0108249 all_source_dois:50 - 10.1021/acs.jcim.3c0108251 zenodo_doi: 10.5281/zenodo.2079402752 curators: []53 promoter: Louis-Félix Nothias54 sponsor: CNRS & Université Côte d'Azur55---5657# Benchmarking and Comparative Performance Evaluation5859## Summary6061Systematically compare a novel method (MIST-CF) against established baseline approaches (SIRIUS, FFN, Xformer) on standardized benchmark datasets using ranking accuracy metrics (top-1, top-3, top-k). This skill enables quantitative assessment of whether a new chemical formula inference model outperforms existing tools on retrospective and prospective MS/MS data.6263## When to use6465Apply this skill when you have trained a new machine learning model for chemical formula or adduct assignment from MS/MS spectra and need to assess whether it offers genuine performance gains over established baselines. Use it specifically when you have access to ground-truth annotations (e.g., NPLIB1, NIST20, CASMI challenge datasets) and must justify publication or deployment claims with comparative accuracy metrics.6667## When NOT to use6869- Input is a single spectrum or small set without ground-truth annotations — use case-by-case evaluation instead.70- Baseline tool (e.g., SIRIUS) is not available or cannot be run on your hardware/OS — benchmarking requires all competing methods executable in the same environment.71- You are comparing only internal model variants (e.g., ablation study) without a published external baseline — use internal ablation evaluation rather than full benchmarking.7273## Inputs7475- MS/MS spectra in MGF format with precursor m/z values76- Ground-truth chemical formula and adduct type annotations77- Candidate formula lists (generated by SIRIUS decomp or chemical enumeration)78- Trained neural network weights for MIST-CF, FFN, Xformer baselines79- Preprocessed spectral data (normalized intensities, noise-filtered peaks)8081## Outputs8283- Per-spectrum ranking predictions (ranked list of candidate formulas with adduct types)84- Top-1, top-3, top-k accuracy metrics for formula assignment85- Top-1, top-3, top-k accuracy metrics for adduct assignment86- Performance summary table (model × accuracy metric)87- Ranking accuracy plots (e.g., cumulative accuracy vs. rank position)88- Comparative analysis report (MIST-CF vs. SIRIUS, FFN, Xformer)8990## How to apply9192Run inference predictions from all competing models (MIST-CF, SIRIUS decomp, FFN, Xformer) on the same test dataset, ensuring identical MS/MS preprocessing (intensity normalization, noise filtering, precursor m/z extraction). For each spectrum, compute ranked predictions for chemical formula and adduct type; evaluate using ranking metrics (top-1 accuracy: first ranked candidate matches ground truth; top-3, top-k: ground truth appears in top k predictions). Tabulate accuracy results side-by-side and generate ranking accuracy plots showing cumulative performance across the test set. On proprietary datasets (e.g., NIST20), train on disjoint splits; on public benchmarks (NPLIB1, CASMI22), use published train-test boundaries. Document hyperparameter tuning (via hyperopt or grid search) for all models on validation data to ensure fair comparison.9394## Related tools9596- **MIST-CF** (Novel energy-based neural network model for ranking chemical formulas and adduct types from MS/MS spectra; main method under evaluation.) — https://github.com/samgoldman97/mist-cf97- **SIRIUS** (Baseline tool for formula enumeration via dynamic programming algorithm (SIRIUS decomp); generates candidate formula lists and provides ranking predictions for comparison.) — https://bio.informatik.uni-jena.de/software/sirius/98- **FFN** (Feed-forward neural network baseline model for formula ranking; included in comparative benchmarking experiments.)99- **Xformer** (Transformer baseline model for formula ranking; included in comparative benchmarking experiments.)100- **SCARF** (Related work on sinusoidal formula embeddings; used as feature representation in MIST-CF architecture.) — https://arxiv.org/abs/2303.06470101102## Examples103104```105. run_scripts/benchmarking/train_mist_cf.sh && . run_scripts/benchmarking/train_ffn.sh && . run_scripts/benchmarking/train_xformer.sh && python run_scripts/benchmarking/eval_models.py106```107108## Evaluation signals109110- Top-k accuracy values are monotonically non-decreasing as k increases (top-1 ≤ top-3 ≤ top-5, etc.).111- MIST-CF top-k accuracy meets or exceeds SIRIUS baseline on retrospective (NPLIB1, NIST20) and prospective (CASMI22) test sets, demonstrating improvement over fragmentation-tree-based method.112- Performance metrics are computed on a consistent test split with no data leakage between training and evaluation sets; hyperparameter tuning used only validation data.113- Adduct assignment accuracy is reported separately and is non-trivial (>baseline random selection of adduct types).114- Comparison tables show that all models were tuned to convergence (no model has obviously suboptimal hyperparameters relative to published results or validation curves).115116## Limitations117118- MIST-CF trained on public NPLIB1 dataset may be less performant than NIST20-trained model (particularly for Orbitrap or higher-resolution MS/MS data), limiting generalizability claims.119- Benchmarking focuses on positive-mode MS/MS only; adduct types are still limited beyond [M+H]+, so evaluation does not cover negative-mode or diverse adduct scenarios.120- SIRIUS baseline still required for initial formula enumeration (candidate generation); MIST-CF learns ranking but does not replace formula discovery, limiting end-to-end de novo capability.121- Prospective evaluation (CASMI22) is a single challenge dataset; retrospective benchmarking on NPLIB1 may overfit to that data distribution.122123## Evidence124125- [other] How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS data?: "research_question: How does MIST-CF's end-to-end energy-based formula transformer perform on ranking chemical formula and adduct assignments compared to SIRIUS-based approaches on benchmark MS/MS"126- [other] Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions.: "4. Evaluate ranking performance by computing top-1, top-3, and top-k accuracy metrics for formula and adduct predictions."127- [other] Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset.: "5. Compare MIST-CF top-k accuracy results against SIRIUS baseline predictions on the same dataset."128- [intro] MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases: "MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases"129- [intro] Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion: "Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion"130- [readme] Retrospective benchmarking: Benchmark MIST-CF performance with baseline models.: "Retrospective benchmarking131Benchmark MIST-CF performance with baseline models."132- [readme] Prospective analysis on CASMI-2022: Compare MIST-CF and SIRIUS on [CASMI-2022](http://www.casmi-contest.org/2022/index.shtml).: "Prospective analysis: CASMI 2022133Compare MIST-CF and SIRIUS on [CASMI-2022](http://www.casmi-contest.org/2022/index.shtml)."134- [readme] This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data).: "This model may be less performant than the model trained on the commercial NIST20 Library (particularly for Orbitrap or higher resolution data)."135- [intro] Considering multiple adduct types beyond [M+H]+ (still only positive mode): "Considering multiple adduct types beyond [M+H]+ (still only positive mode)"