Annotation Benchmark Performance Evaluation

annotation-benchmark-performance-evaluation

Summary

Quantify the quality of untargeted LC-MS metabolite annotation workflows by computing precision, recall, and F1-score against curated reference peak benchmarks. This skill applies when you have both annotated peak predictions and a validated reference dataset, and need to assess whether the annotation pipeline recovers true metabolites and avoids false assignments.

When to use

Apply this skill after running an end-to-end annotation workflow (matching, clustering, filtering, and prioritization) on untargeted LC-MS peak tables, when you have access to a curated reference dataset (df.Ref) containing validated peak assignments for the same biological sample. Use it to quantify top-K annotation performance (e.g., top-3 candidates) and verify that the pipeline achieves acceptable precision and recall before deploying it on new cohorts.

When NOT to use

• The annotation workflow is still incomplete (e.g., clustering and filtering not yet applied)—evaluation requires fully prioritized results.
• No validated reference dataset exists for the sample—performance metrics cannot be computed without ground truth.
• Input is raw mass spectrometry data or a feature table before annotation—you need ranked candidates first.

Inputs

• final ranked annotation table from mWISE.annotation wrapper (with candidate metabolites ordered by diffusion score)
• reference benchmark data frame (df.Ref) containing validated peak identities for the sample
• top.cmps parameter (integer: number of top candidates per peak to evaluate, e.g., 3)

Outputs

• performance metrics table containing precision, recall, and F1-score
• per-peak evaluation results showing which reference peaks were recovered and which predictions were false positives

How to apply

Execute the performanceEvaluation function on the final ranked annotation results, passing the benchmark reference data frame (df.Ref) and specifying the top.cmps argument to define how many top-ranked candidates per peak are considered correct. The function compares predicted annotations against reference peaks and computes three metrics: precision (fraction of predicted annotations matching reference), recall (fraction of reference peaks recovered by predictions), and F1-score (harmonic mean balancing precision and recall). Examine whether performance meets domain expectations—typically precision and recall both > 0.7 for metabolomics—before proceeding. If performance is poor, backtrack to the clustering or diffusion prioritization stages to adjust thresholds (e.g., frequency cutoffs, probability scoring parameters) and re-evaluate iteratively.

Related tools

• mWISE (Provides the performanceEvaluation function and the complete annotation workflow context (matching, clustering, diffusion prioritization) whose output is evaluated) — https://dev.b2s.club/b2slab/mWISE
• R (Language and environment for executing performanceEvaluation and manipulating benchmark and result data frames)

Examples

performanceEvaluation(results = finalAnnotations, df.Ref = df.Ref, top.cmps = 3)

Evaluation signals

• Precision and recall values both lie in [0, 1] and sum-weighted averages are consistent with F1-score (harmonic mean).
• F1-score is ≥ 0.7 (typical threshold for acceptable metabolomics annotation quality) and does not degrade when top.cmps increases beyond optimal rank.
• Comparison of per-peak recovery: all reference peaks in df.Ref appear in the final ranked results, with no unexplained omissions.
• Precision does not decline unexpectedly as top.cmps increases, indicating stable candidate ranking and no artificial inflation from low-confidence predictions.
• Performance metrics are reproducible across multiple runs and are sensitive to changes in upstream parameters (e.g., diffusion input probability vs. binary scoring, cluster frequency thresholds).

Limitations

• Performance depends critically on reference dataset quality and completeness; incomplete or noisy reference peaks (df.Ref) will inflate false-positive and false-negative rates.
• Evaluation is restricted to the top K candidates (top.cmps); if the true metabolite is ranked lower than K, it will not be counted as a recovery, even if present in the final results.
• Precision and recall metrics assume one-to-one matching between predicted and reference peaks; multi-annotation scenarios (one peak matching multiple metabolites, or vice versa) require post-hoc disambiguation logic.
• The Trypanosoma dataset used to demonstrate the skill is negative-mode LC-MS; performance on positive-mode or other ionization regimes may differ significantly.

Evidence

• [other] The performanceEvaluation function computes performance metrics using the benchmark data frame df.Ref, with the top.cmps argument defining the top K candidates considered for the evaluation.: "The performanceEvaluation function computes performance metrics using the benchmark data frame df.Ref, with the top.cmps argument defining the top K candidates considered for the evaluation."
• [other] Evaluate annotation quality by computing top-3 performance metrics (precision, recall, F1-score) against df.Ref benchmark using performanceEvaluation function.: "Evaluate annotation quality by computing top-3 performance metrics (precision, recall, F1-score) against df.Ref benchmark using performanceEvaluation function."
• [intro] mWISE integrates several strategies to provide a fast annotation of peak-intensity tables through matching, clustering, filtering, and diffusion prioritization.: "mWISE integrates several strategies to provide a fast annotation of peak-intensity tables. It consists of three main steps aimed at i) matching mass-to-charge ratio values to KEGG database, ii)"
• [intro] untargeted LC-MS data annotation is a major bottleneck in computational metabolomics that requires validation.: "Several computational strategies have been proposed to overcome untargeted LC-MS data annotation, which is still considered a major bottleneck."