# Top N Accuracy And Auc Metric Calculation

> Use when you have a ranked candidate list (e.g., BGCs sorted by IOKR or strain-correlation score) for each test spectrum, a known ground-truth BGC for each spectrum, and you want to measure retrieval performance across multiple recall depths (top-1 through top-200) and overall discrimination.

- Skill: `holobiomicslab/top-n-accuracy-and-auc-metric-calculation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/top-n-accuracy-and-auc-metric-calculation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/top-n-accuracy-and-auc-metric-calculation/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/top-n-accuracy-and-auc-metric-calculation

---


# top-n-accuracy-and-auc-metric-calculation

## Summary

Compute ranking-based accuracy metrics (top-1, top-5, top-10, etc.) and area-under-curve (AUC) to evaluate how often a correct BGC-spectrum match ranks within the top n candidates, and assess discriminative power against a randomized baseline. This skill is essential for ranking-based retrieval tasks where a single ground-truth target must be re-identified within a sorted candidate list.

## When to use

You have a ranked candidate list (e.g., BGCs sorted by IOKR or strain-correlation score) for each test spectrum, a known ground-truth BGC for each spectrum, and you want to measure retrieval performance across multiple recall depths (top-1 through top-200) and overall discrimination ability (AUC). This is appropriate when you cannot threshold on a single decision boundary and instead need to understand how ranking quality scales with acceptance of more candidates.

## When NOT to use

- Input is a probabilistic classifier with soft confidence scores rather than deterministic ranks—use calibration and threshold-tuning instead.
- The ranking pool is smaller than ~50 candidates or contains many ties at the same score—top-n accuracy becomes unstable and AUC may not be interpretable.
- Ground truth is ambiguous (multiple equally valid BGCs per spectrum)—AUC computation requires binary labels and will produce spurious results.

## Inputs

- ranked BGC candidate list per spectrum (spectrum ID, candidate BGC ID, score)
- ground-truth BGC-spectrum pairs (2966 validated links)
- candidate BGC pool (2242 BGCs with MIBiG homology structure assignments)

## Outputs

- top-n accuracy table (n ∈ {1, 5, 10, 20, 200})
- AUC score and 95% confidence interval
- rank distribution histogram (with validated link positions overlay)
- p-value comparing validated vs. all-link score distributions

## How to apply

For each of the 2966 BGC-spectrum pairs, rank all candidate BGCs (2242 with MIBiG structure assignments) by descending score. Record the rank position of the correct BGC. For each threshold n ∈ {1, 5, 10, 20, 200}, compute top-n accuracy as the proportion of spectra for which the correct BGC appears in positions 1 to n. For AUC, use the rank positions to construct a binary classification problem: for each spectrum, assign a positive label to the correct BGC and negative labels to all other ranked candidates, then compute area-under-the-receiver-operating-characteristic curve. Generate a randomized baseline by permuting BGC ranks for each spectrum independently and repeating the same calculations; valid IOKR performance must substantially exceed baseline AUC (0.5209 baseline vs. 0.6534 observed). Report mean ± variance of scores, p-values comparing validated vs. all-link distributions, and visual histograms overlaying validated-link positions on the full score distribution.

## Related tools

- **GNPS** (source of MS2 spectra (6246 test spectra) matched to MIBiG BGCs via InChIKey for ground-truth pair construction)
- **MIBiG** (source of BGC candidate pool (2242 with structure assignments) and ground-truth validated BGC-spectrum pairs)
- **NPLinker** (framework orchestrating rank computation and metric calculation over BGC-spectrum pairs) — https://github.com/sdrogers/nplinker

## Evaluation signals

- Top-n accuracy must be monotonically non-decreasing with increasing n (e.g., top-1 ≤ top-5 ≤ top-10); any decrease indicates a computational error.
- AUC must fall in [0.5, 1.0]; AUC > 0.5 indicates the scoring function ranks correct BGCs higher than random; AUC = 0.5 exactly suggests no discriminative power.
- Validated-link score distribution must have significantly higher mean than all-link distribution (p < 0.05), with visual separation in histograms.
- Randomized baseline AUC must be approximately 0.5 (±0.01); deviations suggest incorrect rank permutation or label assignment.
- Reported results must match published Table 3 and Figure S2 (IOKR top-1: 0.1208, top-5: 0.1708, AUC: 0.6534; baseline top-1: 0.0, top-5: 0.0014, AUC: 0.5209).

## Limitations

- Performance breakdown by natural product compound class is difficult due to insufficient test set size for statistical stratification.
- IOKR applicability is restricted to BGCs with considerable homology to MIBiG entries; low-homology BGCs are not included in the candidate pool and cannot be ranked.
- Top-n accuracy is sensitive to candidate pool size and composition; results are not directly comparable across datasets with different numbers of structure-annotated BGCs.
- AUC computation assumes all negative candidates are equally undesirable; it does not account for semantic similarity among non-matching BGCs (e.g., same biosynthetic family).

## Evidence

- [results] IOKR top-1 accuracy: 0.1208; top-5: 0.1708; top-10: 0.1870; top-20: 0.2121; top-200: 0.2946; AUC: 0.6534: "Table 3 shows the top-n performance of IOKR, i.e. how often the 'true' BGC match for a given spectrum is among the top n matches returned by IOKR: top-1: 0.1208, top-5: 0.1708, top-10: 0.1870"
- [results] random baseline top-1 accuracy: 0.0; top-5: 0.0014; top-10: 0.0044; top-20: 0.0103; top-200: 0.1486; AUC: 0.5209: "baseline score was estimated by randomising the rank of the structures for each spectrum, and the same process was repeated: top-1: 0.0, top-5: 0.0014, top-10: 0.0044, top-20: 0.0103, top-200:"
- [other] Calculate top-n accuracy (n=1,5,10,20,200) and area-under-curve (AUC) by comparing the rank of the correct BGC relative to all ranked candidates for each spectrum: "Calculate top-n accuracy (n=1,5,10,20,200) and area-under-curve (AUC) by comparing the rank of the correct BGC relative to all ranked candidates for each spectrum, and compare against a randomized"
- [other] IOKR mean score of 0.0105 for all 2966 BGC-spectrum links and 0.0364 for validated links (p=1.7968 × 10−9): "IOKR achieves a mean score of 0.0105 for all 2966 BGC-spectrum links and 0.0364 for validated links (p=1.7968 × 10−9)"
- [other] Construct 2966 BGC-spectrum pairs by matching MIBiG entries to GNPS spectra using the first part of the InChIKey: "Construct 2966 BGC-spectrum pairs by matching MIBiG entries to GNPS spectra using the first part of the InChIKey to avoid confounding by stereoisomerism."