# Formula Ranking Accuracy Evaluation

> Use when use this skill after training or fine-tuning a chemical formula transformer model on annotated tandem MS/MS spectra, when you need to measure whether the model's ranked formula candidates match ground truth.

- Skill: `holobiomicslab/formula-ranking-accuracy-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/formula-ranking-accuracy-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/formula-ranking-accuracy-evaluation/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/formula-ranking-accuracy-evaluation

---


# formula-ranking-accuracy-evaluation

## Summary

Evaluate the ranking quality of predicted chemical formulas and adduct assignments by computing top-k accuracy metrics (top-1, top-5, top-10) on held-out test spectra with annotated ground-truth formulas. This skill verifies whether a formula inference model (e.g., MIST-CF) correctly places the true molecular formula within the top-k candidates ranked by model confidence.

## When to use

Use this skill after training or fine-tuning a chemical formula transformer model on annotated tandem MS/MS spectra, when you need to measure whether the model's ranked formula candidates match ground truth. Apply it on a held-out test set to avoid overfitting bias, and especially when extending the model to a new ionisation mode (e.g., negative-mode adducts) or adduct type where no prior validation exists.

## When NOT to use

- Test set is not held-out or overlaps with training data—this will produce inflated accuracy estimates and fail to detect overfitting.
- Annotated formulas in test set are absent or ambiguous—ground truth is required to define a hit.
- Model has not been trained or fine-tuned on the target ionisation mode or adduct type—baseline accuracy will be near zero and not interpretable as a real performance limitation.

## Inputs

- held-out test set of annotated MS/MS spectra (MGF format or equivalent) with ground-truth molecular formulas and adduct assignments
- trained formula inference model (checkpoint or serialized neural network weights)
- ranked formula candidate lists output by the model, with confidence scores or energy-based model scores

## Outputs

- metrics table with top-1, top-5, and top-10 formula ranking accuracy per adduct type
- accuracy scores aggregated across all test spectra and stratified by adduct class
- failure analysis or confusion matrix identifying which formulas or adducts are most frequently misranked

## How to apply

Split your annotated MS/MS spectra into training and held-out test sets. Run the formula inference model (e.g., MIST-CF) on each test spectrum to generate a ranked list of candidate formulas with associated adduct assignments. For each spectrum, check whether the annotated ground-truth formula appears in the top-1, top-5, and top-10 positions of the ranked output. Compute the fraction of spectra where the true formula was recovered at each rank cutoff, storing results in a metrics table indexed by adduct type (e.g., [M-H]−, [M+Cl]−, [M+FA]−) to enable cross-adduct comparison and identify performance bottlenecks.

## Related tools

- **MIST-CF** (formula transformer model that generates ranked candidate formulas and adduct assignments from MS/MS spectra; primary inference engine being evaluated) — https://github.com/samgoldman97/mist-cf
- **SCARF** (provides sinusoidal formula embeddings used as input representation in MIST-CF formula transformer, maintaining consistency across formula tokenization)
- **SIRIUS** (used for deterministic formula enumeration (SIRIUS decomp) to generate candidate formula lists; baseline method for comparison in benchmarking experiments) — https://bio.informatik.uni-jena.de/software/sirius/

## Evaluation signals

- Top-1 accuracy should be >0 for each adduct type; very low values (<5%) indicate model underfitting or data preprocessing errors.
- Top-1 accuracy should be substantially higher than random baseline (which depends on formula candidate space size); if top-1 ≈ 1/|candidate set|, the model is not learning.
- Accuracy should improve monotonically: top-1 ≤ top-5 ≤ top-10; violation indicates output ranking inconsistency or data leakage.
- Performance should be stratified and comparable across adduct types if training data is balanced; large disparities (e.g., >10% gap between [M-H]− and [M+Cl]−) suggest adduct-specific underfitting or dataset skew.
- Metrics table should be fully populated with no missing or NaN values; check for incomplete predictions or filtering that silently drops test spectra.

## Limitations

- Model-dependent performance: ranking accuracy reflects both the formula inference architecture and the quality of MS/MS spectra preprocessing (noise removal, intensity normalization); poor spectra will yield artificially low accuracy even with a well-trained model.
- Adduct assumption: evaluation assumes the true adduct type is known and present in the model's tokenization layer; if an observed spectrum is [M+Na]+ but the model only considers [M+H]+ and [M+NH4]+, accuracy will be zero regardless of correct molecular formula rank.
- Test set annotation quality: errors or ambiguities in ground-truth formula or adduct labels introduce noise; high-confidence annotations from validated repositories (MassIVE, MetaboLights) are essential.
- Generalization gap: models trained on NPLIB1 or NIST20 may show degraded accuracy on out-of-distribution spectra (e.g., different instruments, resolution, or compound classes); separate evaluation on prospective datasets (CASMI 2022) recommended to assess real-world utility.

## Evidence

- [other] Evaluate the fine-tuned model on a held-out negative-mode test set by computing top-1, top-5, and top-10 formula ranking accuracy: "Evaluate the fine-tuned model on a held-out negative-mode test set by computing top-1, top-5, and top-10 formula ranking accuracy."
- [intro] MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases: "MIST-CF ranks chemical formula and adduct assignments for an unknown mass spectrum using an end-to-end energy based modeling approach, without referencing any spectrum databases"
- [other] Compile results in a metrics table comparing performance across negative adduct types: "Compile results in a metrics table comparing performance across negative adduct types."
- [intro] MIST-CF considers multiple adduct types beyond [M+H]+ (still only positive mode): "Considering multiple adduct types beyond [M+H]+ (still only positive mode)"
- [intro] Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion: "Instead of computing fragmentation trees, MIST-CF adopts a formula transformer neural network architecture and learns in a data dependent fashion"