Molecular Structure Ground Truth Matching And Accuracy Evaluation

molecular-structure-ground-truth-matching-and-accuracy-evaluation

Summary

Compares predicted molecular structures (formula and connectivity) against experimentally known ground-truth structures to compute exact recovery rates and other fidelity metrics. This skill is essential for validating end-to-end NMR-to-structure models on molecules up to 19 heavy atoms.

When to use

After an end-to-end neural model (CNN + transformer) has generated predicted molecular structures (formula and connectivity) from 1D NMR spectra. Use this skill to quantify how many predictions exactly match the experimentally determined ground-truth structures in a held-out test set, yielding top-k exact structure recovery rates that reflect real-world accuracy.

When NOT to use

• When predicted structures have not yet been generated by the model; this skill is a validation step, not a prediction step.
• When comparing spectra or spectral features directly without intermediate structure prediction—use spectral similarity metrics instead.
• When ground-truth structures are not available or are unreliable; accuracy evaluation requires experimentally confirmed reference structures.

Inputs

• predicted molecular structures (SMILES strings or connectivity graphs) from the end-to-end model
• ground-truth molecular structures (canonical SMILES or adjacency matrices) for test molecules
• test set metadata (molecule identifiers, heavy atom counts ≤19)
• model confidence scores or ranking for top-k evaluation

Outputs

• top-k exact structure recovery rates (e.g., top-1, top-5, top-10 accuracy percentages)
• count of exact matches and mismatches per test molecule
• accuracy metrics stratified by molecular size (heavy atom counts)
• detailed match/mismatch report for error analysis

How to apply

Load the ground-truth molecular structures (canonical SMILES or connectivity graphs) for the test set molecules (≤19 heavy atoms). For each predicted structure output by the model, perform a canonical comparison (e.g., SMILES normalization or graph isomorphism) against its corresponding ground-truth structure. Count exact matches (both molecular formula and connectivity must match) and compute top-k metrics by ranking predictions by model confidence. Report the fraction of test molecules where the top-1, top-5, or top-10 prediction contains the correct structure. Cross-validate that the ground-truth set is disjoint from training and validation sets to avoid inflated metrics.

Related tools

• convolutional neural network (encodes raw 1D NMR spectra into learned feature representations upstream of structure prediction)
• transformer architecture (assembles spectral features and molecular fragments into connectivity predictions that are then matched against ground truth)

Evaluation signals

• Top-k accuracy metrics (top-1, top-5, top-10) are computed and match or exceed the paper's reported baseline performance on the same test set.
• Exact matches are verified by canonical SMILES comparison or graph isomorphism to exclude false positives from different but equivalent structural representations.
• Ground-truth test set is confirmed to be disjoint from training and validation splits (no data leakage).
• Accuracy is stratified by molecular size (heavy atom bins) and shows consistent recovery rates across the ≤19 heavy atom range reported in the paper.
• Mismatch cases are inspectable: predicted vs. ground-truth structures can be visually inspected or dumped to a report for error classification (e.g., wrong formula, correct formula but wrong connectivity).

Limitations

• The skill is limited to test molecules with ≤19 heavy atoms, as the model was trained and validated only in this regime; extrapolation to larger molecules is not supported by the paper.
• Accuracy depends on the quality and completeness of the ground-truth reference structures; experimental errors or missing reference data will bias metrics.
• Top-k metrics assume the model outputs a ranked list of candidate structures; if only a single prediction is returned, only top-1 accuracy can be computed.
• The skill does not account for stereoisomerism or 3D conformation; it evaluates connectivity and formula only, so molecules differing only in stereochemistry will be treated as different structures.

Evidence

• [intro] end-to-end structure recovery accuracy metrics: "What are the exact structure recovery accuracy metrics achieved by the end-to-end NMR2Struct model on test molecules containing up to 19 heavy atoms?"
• [intro] multitask prediction generates formula and connectivity: "Perform multitask prediction to generate both molecular formula and connectivity output."
• [intro] top-k comparison against ground truth: "Compute top-k exact structure recovery rates by comparing predicted structures against ground-truth molecular structures."
• [intro] model outputs structure from spectra: "predicts the molecular structure (formula and connectivity) of an unknown compound solely based on its 1D 1H and/or 13C NMR spectra"
• [intro] scope: molecules up to 19 heavy atoms: "We demonstrate the effectiveness of this framework on molecules with up to 19 heavy (non-hydrogen) atoms"