HolobiomicsLab ms-modification-site-evaluation
Summary
Quantitatively evaluate the accuracy of predicted modification sites on unknown compounds by comparing ModiFinder-generated probability distributions against known structural modifications using BasicEvaluationEngine scoring metrics (is_max and average_distance). This skill validates whether the predicted site probabilities correctly localize the actual modification.
When to use
After ModiFinder has generated modification site probability scores for an unknown compound by comparing its MS/MS spectrum to a known analog, and you have access to the true structure of the unknown compound (oracle mode) or a reference modification site annotation. Use this skill to quantify whether the prediction succeeded—i.e., whether the highest-probability site(s) align with the true modification location.
When NOT to use
- The modified compound structure is unknown and oracle-mode refinement has not been applied; evaluation requires ground truth.
- ModiFinder probabilities have not yet been generated; evaluation requires a complete probability distribution over modification sites.
- You are only interested in visual inspection of the alignment and do not need quantitative validation.
Inputs
- analog_structure (RDKit molecule object representing the known compound)
- target_structure (RDKit molecule object representing the modified/unknown compound)
- modification_probabilities (numpy array or dict mapping atoms to probability scores, generated by ModiFinder.generate_probabilities())
- evaluation_method (string: 'is_max' or 'average_distance')
Outputs
- evaluation_score (float: 1.0 or 0.0 for is_max; 0.0–1.0 for average_distance)
- evaluation_results (dictionary or JSON object containing score, method, and optionally the input probability vector)
How to apply
Instantiate BasicEvaluationEngine with the desired scoring method (either 'is_max' to test whether the true site received the maximum probability, or 'average_distance' to measure the average atom-level distance from predicted high-probability atoms to the true modification site). Call evaluate_single() with the known/analog structure, modified/unknown structure, and the ModiFinder-generated probabilities. The is_max score (binary: 0 or 1) succeeds if the true site is correctly ranked highest; average_distance (continuous: 0–1, lower is better) provides sensitivity to near-misses by penalizing predictions that are structurally close but not exact. Compare returned scores against project baselines (e.g., is_max=1.0 and average_distance=0.514 for ModiFinder under default Modified Cosine + MAGMa) to assess whether your configuration matches expected performance.
Related tools
- ModiFinder (Generates modification site probability distributions via generate_probabilities() that are passed to BasicEvaluationEngine for scoring) — https://github.com/Wang-Bioinformatics-Lab/ModiFinder_base
- BasicEvaluationEngine (Computes is_max and average_distance evaluation metrics by comparing predicted probabilities against known structures) — https://github.com/Wang-Bioinformatics-Lab/ModiFinder_base
- RDKit (Provides molecule representation and structure comparison primitives used by BasicEvaluationEngine) — http://www.rdkit.org/
- Python (Required runtime for ModiFinder and BasicEvaluationEngine (≥3.9))
Examples
from modifinder import BasicEvaluationEngine; engine = BasicEvaluationEngine(default_method='average_distance'); score = engine.evaluate_single(analog_structure, target_structure, probabilities); print(f'average_distance={score}')
Evaluation signals
- is_max score equals 1.0 if and only if the true modification site receives the maximum probability in the predicted distribution.
- average_distance score is in the range [0, 1] with lower values indicating predictions closer to the true modification site; baseline of 0.514 for ModiFinder default configuration provides a reference benchmark.
- Consistency check: is_max=1.0 should generally yield a low average_distance (since the true site is ranked first); if both metrics are available, inspect their correlation.
- JSON output includes the input probability vector to enable post-hoc tracing and visualization of which atoms received high confidence.
- Evaluation score reproducibility: re-running evaluate_single() with identical inputs should yield identical scores (deterministic).
Limitations
- Evaluation requires knowledge of the true modification site; if the ground truth structure is incorrect or ambiguous, the score may misleadingly penalize correct predictions.
- average_distance depends on the choice of atom-alignment algorithm used by BasicEvaluationEngine; different structure normalization or graph matching strategies could yield different scores.
- is_max is binary and does not reward near-correct predictions; average_distance is more forgiving but can mask failure modes where multiple high-probability atoms are far from the truth.
- The skill does not validate the quality of the input spectra, tolerances (mz_tolerance=0.01, ppm_tolerance=40, ratio_to_base_peak=0.01), or ModiFinder configuration; poor upstream choices will lead to poor scores that do not reflect a failure of evaluation itself.
Evidence
- [other] Instantiate BasicEvaluationEngine with default_method='average_distance': "Instantiate BasicEvaluationEngine with default_method='average_distance'"
- [other] Call evaluate_single() with analog and target structures and probabilities: "Call evaluate_single(analog_structure, target_structure, probabilities) to compute average_distance score; call again with evaluation_method='is_max' to compute is_max score"
- [other] ModiFinder baseline scores: is_max=1.0 and average_distance=0.514: "ModiFinder achieves BasicEvaluationEngine scores of is_max = 1.0 and average_distance = 0.514 under the default Modified Cosine + MAGMa condition"
- [other] Baseline comparison to validate configuration: "Compare the returned score against the reference value of 0.514 and document the result"
- [other] Output includes probability vector for tracing: "Write both scores and intermediate probability vector to evaluation_results.json"