# Annotation Benchmark Performance Evaluation

> Use when after running an end-to-end annotation workflow (matching, clustering, filtering, and prioritization) on untargeted LC-MS peak tables, when you have access to a curated reference dataset (df.Ref) containing validated peak assignments for the same biological sample.

- Skill: `holobiomicslab/annotation-benchmark-performance-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/annotation-benchmark-performance-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/annotation-benchmark-performance-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/annotation-benchmark-performance-evaluation

---


# annotation-benchmark-performance-evaluation

## Summary

Quantify the quality of untargeted LC-MS metabolite annotation workflows by computing precision, recall, and F1-score against curated reference peak benchmarks. This skill applies when you have both annotated peak predictions and a validated reference dataset, and need to assess whether the annotation pipeline recovers true metabolites and avoids false assignments.

## When to use

Apply this skill after running an end-to-end annotation workflow (matching, clustering, filtering, and prioritization) on untargeted LC-MS peak tables, when you have access to a curated reference dataset (df.Ref) containing validated peak assignments for the same biological sample. Use it to quantify top-K annotation performance (e.g., top-3 candidates) and verify that the pipeline achieves acceptable precision and recall before deploying it on new cohorts.

## When NOT to use

- The annotation workflow is still incomplete (e.g., clustering and filtering not yet applied)—evaluation requires fully prioritized results.
- No validated reference dataset exists for the sample—performance metrics cannot be computed without ground truth.
- Input is raw mass spectrometry data or a feature table before annotation—you need ranked candidates first.

## Inputs

- final ranked annotation table from mWISE.annotation wrapper (with candidate metabolites ordered by diffusion score)
- reference benchmark data frame (df.Ref) containing validated peak identities for the sample
- top.cmps parameter (integer: number of top candidates per peak to evaluate, e.g., 3)

## Outputs

- performance metrics table containing precision, recall, and F1-score
- per-peak evaluation results showing which reference peaks were recovered and which predictions were false positives

## How to apply

Execute the performanceEvaluation function on the final ranked annotation results, passing the benchmark reference data frame (df.Ref) and specifying the top.cmps argument to define how many top-ranked candidates per peak are considered correct. The function compares predicted annotations against reference peaks and computes three metrics: precision (fraction of predicted annotations matching reference), recall (fraction of reference peaks recovered by predictions), and F1-score (harmonic mean balancing precision and recall). Examine whether performance meets domain expectations—typically precision and recall both > 0.7 for metabolomics—before proceeding. If performance is poor, backtrack to the clustering or diffusion prioritization stages to adjust thresholds (e.g., frequency cutoffs, probability scoring parameters) and re-evaluate iteratively.

## Related tools

- **mWISE** (Provides the performanceEvaluation function and the complete annotation workflow context (matching, clustering, diffusion prioritization) whose output is evaluated) — https://dev.b2s.club/b2slab/mWISE
- **R** (Language and environment for executing performanceEvaluation and manipulating benchmark and result data frames)

## Examples

```
performanceEvaluation(results = finalAnnotations, df.Ref = df.Ref, top.cmps = 3)
```

## Evaluation signals

- Precision and recall values both lie in [0, 1] and sum-weighted averages are consistent with F1-score (harmonic mean).
- F1-score is ≥ 0.7 (typical threshold for acceptable metabolomics annotation quality) and does not degrade when top.cmps increases beyond optimal rank.
- Comparison of per-peak recovery: all reference peaks in df.Ref appear in the final ranked results, with no unexplained omissions.
- Precision does not decline unexpectedly as top.cmps increases, indicating stable candidate ranking and no artificial inflation from low-confidence predictions.
- Performance metrics are reproducible across multiple runs and are sensitive to changes in upstream parameters (e.g., diffusion input probability vs. binary scoring, cluster frequency thresholds).

## Limitations

- Performance depends critically on reference dataset quality and completeness; incomplete or noisy reference peaks (df.Ref) will inflate false-positive and false-negative rates.
- Evaluation is restricted to the top K candidates (top.cmps); if the true metabolite is ranked lower than K, it will not be counted as a recovery, even if present in the final results.
- Precision and recall metrics assume one-to-one matching between predicted and reference peaks; multi-annotation scenarios (one peak matching multiple metabolites, or vice versa) require post-hoc disambiguation logic.
- The Trypanosoma dataset used to demonstrate the skill is negative-mode LC-MS; performance on positive-mode or other ionization regimes may differ significantly.

## Evidence

- [other] The performanceEvaluation function computes performance metrics using the benchmark data frame df.Ref, with the top.cmps argument defining the top K candidates considered for the evaluation.: "The performanceEvaluation function computes performance metrics using the benchmark data frame df.Ref, with the top.cmps argument defining the top K candidates considered for the evaluation."
- [other] Evaluate annotation quality by computing top-3 performance metrics (precision, recall, F1-score) against df.Ref benchmark using performanceEvaluation function.: "Evaluate annotation quality by computing top-3 performance metrics (precision, recall, F1-score) against df.Ref benchmark using performanceEvaluation function."
- [intro] mWISE integrates several strategies to provide a fast annotation of peak-intensity tables through matching, clustering, filtering, and diffusion prioritization.: "mWISE integrates several strategies to provide a fast annotation of peak-intensity tables. It consists of three main steps aimed at i) matching mass-to-charge ratio values to KEGG database, ii)"
- [intro] untargeted LC-MS data annotation is a major bottleneck in computational metabolomics that requires validation.: "Several computational strategies have been proposed to overcome untargeted LC-MS data annotation, which is still considered a major bottleneck."