Moleculariq Eval

moleculariq-eval

MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs — Bartmann et al. (2026) (arXiv:2601.15279, 2026)

What this evaluates

Evaluates large language models' ability to perform symbolic reasoning on molecular graphs, including counting atomic features, indexing substructures, and generating constrained molecular structures. It probes whether models understand chemical topology and composition rather than relying on memorized token patterns or canonical SMILES conventions.

Datasets

• MOLECULARIQ — total ?; splits: test (-1); repo https://github.com/ml-jku/moleculariq

Metrics

• accuracy (primary) — range: percent
  • Proportion of instances where the model's output exactly matches the symbolically verified ground truth derived from the molecular graph. Computed as correct predictions divided by total instances.
• type_validity — range: percent
  • Proportion of outputs that conform to the expected data type or format (e.g., valid SMILES string, correct numerical format) regardless of semantic correctness. Used to isolate reasoning failures from extraction artifacts.

Input / output format

Input: Molecular structures provided as SMILES strings (canonical, randomized, kekulized, or ring-enumerated) paired with natural language instructions to count features, index locations, or generate constrained structures.

Output: Numerical counts, structural indices/locations, or generated molecular structures/SMILES, depending on the task type.

Scoring recipe

def score(predictions, gold):
    correct = 0
    valid_type = 0
    for pred, gold in zip(predictions, gold):
        if is_valid_format(pred, gold.expected_type):
            valid_type += 1
        if pred == gold.answer:
            correct += 1
    return {
        'accuracy': correct / len(predictions),
        'type_validity': valid_type / len(predictions)
    }

Common pitfalls

• Models often rely on SMILES canonicalization and token patterns rather than genuine graph reasoning, causing sharp accuracy drops under representation perturbations.
• Evaluation scores can conflate reasoning failures with output formatting errors; the benchmark uses type validity to isolate genuine semantic mistakes.
• Multitask questions require all subtasks to be correct, but prompting multiple subtasks can sometimes improve individual subtask solvability, complicating difficulty scaling.

Evidence (verbatim from paper)

Accuracy drops sharply as constraint-set prevalence (i.e., how frequently the requested features co-occur in PubChem molecules) decreases (Fig. C1) and once three or more constraints are requested (Fig. C12). Table C13 and Fig. C13 show that type validity is high, often 80 - 90% for top models, while accuracy is substantially lower, indicating that most failures are semantically wrong answers, not malformed outputs.

Citation

@misc{bartmann2026moleculariq,
  title={MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs},
  author={Bartmann et al. (2026)},
  year={2026},
  note={arXiv:2601.15279}
}

• arXiv: 2601.15279