HolobiomicsLab annotation-accuracy-and-coverage-metrics-computation
Summary
Compute validation metrics (accuracy and coverage) for molecular structure annotations by comparing pipeline predictions against reference annotations. This skill quantifies the quality of biotransformation-based annotation methods on untargeted metabolomics data.
When to use
Apply this skill after executing an end-to-end structure annotation pipeline (such as BAM) on a validation dataset with known reference annotations. Use it when you need to benchmark annotation performance, verify reproducibility of reported metrics, or assess the quality of predicted molecular structures against ground truth.
When NOT to use
- No reference annotations are available or validation dataset is absent
- Pipeline has not yet been executed on the validation data (metrics require both predictions and ground truth)
- Annotations are not directly comparable (e.g., predictions in different chemical representation formats without conversion)
Inputs
- Pipeline output annotations (predicted structure identifiers, SMILES, or InChI strings)
- Reference annotation dataset (ground-truth molecular structures with SMILES or InChI)
- Query molecules list (suspects/anchors with identifiers and masses)
Outputs
- Annotation accuracy metric (fraction of correct predictions)
- Annotation coverage metric (fraction of queries receiving predictions)
- Metrics report documenting accuracy, coverage, and performance benchmarks
How to apply
Obtain both the predicted structure annotations generated by the annotation pipeline (e.g., BAM output) and the reference annotations from your validation dataset. For each predicted annotation, compare it against the corresponding reference annotation using a metric of structural equivalence (e.g., SMILES matching or InChI comparison). Compute accuracy as the fraction of predicted annotations that match reference annotations; compute coverage as the fraction of query molecules that received a prediction. Generate a metrics report documenting achieved accuracy, coverage, and any additional performance benchmarks (e.g., per-molecule or per-class breakdowns). Document the comparison method and any tie-breaking rules used when multiple candidate structures are ranked.
Related tools
- HassounLab/BAM (Biotransformation-based annotation pipeline that generates predicted molecular structure annotations for validation) — https://github.com/HassounLab/BAM
- HassounLab/PROXIMAL2 (Dependency for BAM; generates biotransformation operators used in structure annotation) — https://github.com/HassounLab/PROXIMAL2
- HassounLab/GNN-SOM (Dependency for BAM; predicts site-of-metabolism for ranking candidate structures) — https://github.com/HassounLab/GNN-SOM
Evaluation signals
- Accuracy is computed as the fraction of predicted annotations matching reference annotations (range 0–1)
- Coverage is computed as the fraction of query molecules that received at least one prediction (range 0–1)
- Metrics report includes per-dataset breakdowns (e.g., KEGG vs. RetroRules reaction data) when applicable
- Comparison method (e.g., SMILES or InChI matching) is explicitly documented to ensure reproducibility
- Results are stratified by molecular class, anchor type, or suspect mass range where data permits
Limitations
- Accuracy depends on the quality and completeness of the reference annotation dataset; incomplete or incorrect ground truth will bias results
- Coverage may be artificially low if the pipeline fails to generate predictions for molecules outside the scope of the reaction rule dataset (e.g., rare biotransformations)
- Structural equivalence comparison requires careful handling of stereochemistry, tautomerism, and chemical representation canonicalization to avoid false mismatches
- Metrics do not capture ranking quality: a correct structure ranked low is counted the same as one ranked high
Evidence
- [other] Compute validation metrics (annotation accuracy, coverage) by comparing pipeline predictions against reference annotations.: "Compute validation metrics (annotation accuracy, coverage) by comparing pipeline predictions against reference annotations."
- [readme] All data necessary to run the evaluation of BAM described in our paper is included in the data folder.: "All data necessary to run the evaluation of BAM described in our paper is included in the data folder."
- [readme] BAM checks if the suspect molecule is known by checking whether the SMILES or InChI is specified in the molecules_of_interest csv file.: "BAM checks if the suspect molecule is known by checking whether the SMILES or InChI is specified in the molecules_of_interest csv file."
- [other] Generate a metrics report documenting achieved accuracy, coverage, and any performance benchmarks.: "Generate a metrics report documenting achieved accuracy, coverage, and any performance benchmarks."