molecular-structure-ground-truth-matching-and-accuracy-evaluation
Summary
Compares predicted molecular structures (formula and connectivity) against experimentally known ground-truth structures to compute exact recovery rates and other fidelity metrics. This skill is essential for validating end-to-end NMR-to-structure models on molecules up to 19 heavy atoms.
When to use
After an end-to-end neural model (CNN + transformer) has generated predicted molecular structures (formula and connectivity) from 1D NMR spectra. Use this skill to quantify how many predictions exactly match the experimentally determined ground-truth structures in a held-out test set, yielding top-k exact structure recovery rates that reflect real-world accuracy.
When NOT to use
- When predicted structures have not yet been generated by the model; this skill is a validation step, not a prediction step.
- When comparing spectra or spectral features directly without intermediate structure prediction—use spectral similarity metrics instead.
- When ground-truth structures are not available or are unreliable; accuracy evaluation requires experimentally confirmed reference structures.
Inputs
- predicted molecular structures (SMILES strings or connectivity graphs) from the end-to-end model
- ground-truth molecular structures (canonical SMILES or adjacency matrices) for test molecules
- test set metadata (molecule identifiers, heavy atom counts ≤19)
- model confidence scores or ranking for top-k evaluation
Outputs
- top-k exact structure recovery rates (e.g., top-1, top-5, top-10 accuracy percentages)
- count of exact matches and mismatches per test molecule
- accuracy metrics stratified by molecular size (heavy atom counts)
- detailed match/mismatch report for error analysis
How to apply
Load the ground-truth molecular structures (canonical SMILES or connectivity graphs) for the test set molecules (≤19 heavy atoms). For each predicted structure output by the model, perform a canonical comparison (e.g., SMILES normalization or graph isomorphism) against its corresponding ground-truth structure. Count exact matches (both molecular formula and connectivity must match) and compute top-k metrics by ranking predictions by model confidence. Report the fraction of test molecules where the top-1, top-5, or top-10 prediction contains the correct structure. Cross-validate that the ground-truth set is disjoint from training and validation sets to avoid inflated metrics.
Related tools
- convolutional neural network (encodes raw 1D NMR spectra into learned feature representations upstream of structure prediction)
- transformer architecture (assembles spectral features and molecular fragments into connectivity predictions that are then matched against ground truth)
Evaluation signals
- Top-k accuracy metrics (top-1, top-5, top-10) are computed and match or exceed the paper's reported baseline performance on the same test set.
- Exact matches are verified by canonical SMILES comparison or graph isomorphism to exclude false positives from different but equivalent structural representations.
- Ground-truth test set is confirmed to be disjoint from training and validation splits (no data leakage).
- Accuracy is stratified by molecular size (heavy atom bins) and shows consistent recovery rates across the ≤19 heavy atom range reported in the paper.
- Mismatch cases are inspectable: predicted vs. ground-truth structures can be visually inspected or dumped to a report for error classification (e.g., wrong formula, correct formula but wrong connectivity).
Limitations
- The skill is limited to test molecules with ≤19 heavy atoms, as the model was trained and validated only in this regime; extrapolation to larger molecules is not supported by the paper.
- Accuracy depends on the quality and completeness of the ground-truth reference structures; experimental errors or missing reference data will bias metrics.
- Top-k metrics assume the model outputs a ranked list of candidate structures; if only a single prediction is returned, only top-1 accuracy can be computed.
- The skill does not account for stereoisomerism or 3D conformation; it evaluates connectivity and formula only, so molecules differing only in stereochemistry will be treated as different structures.
Evidence
- [intro] end-to-end structure recovery accuracy metrics: "What are the exact structure recovery accuracy metrics achieved by the end-to-end NMR2Struct model on test molecules containing up to 19 heavy atoms?"
- [intro] multitask prediction generates formula and connectivity: "Perform multitask prediction to generate both molecular formula and connectivity output."
- [intro] top-k comparison against ground truth: "Compute top-k exact structure recovery rates by comparing predicted structures against ground-truth molecular structures."
- [intro] model outputs structure from spectra: "predicts the molecular structure (formula and connectivity) of an unknown compound solely based on its 1D 1H and/or 13C NMR spectra"
- [intro] scope: molecules up to 19 heavy atoms: "We demonstrate the effectiveness of this framework on molecules with up to 19 heavy (non-hydrogen) atoms"