Moleculeqa Eval

moleculeqa-eval

MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension — Lu et al. (2024) (arXiv:2403.08192, 2024)

What this evaluates

Evaluates factual accuracy and reliability in molecular comprehension by testing whether models correctly describe molecular properties, structures, applications, and sources without hallucination or omission. It probes domain-specific knowledge retention and consistency against authoritative chemical corpora.

Datasets

• MoleculeQA — total 61574; splits: train (49993), dev (5795), test (5786); repo https://github.com/IDEA-XL/MoleculeQA

Metrics

• factual accuracy (primary) — range: [0, 1]
  • Ratio of correct predictions to the total number of evaluated slots. Domain experts classify generated content as correct (matches ground truth), miss (ground truth aspect missing), or error (factual inconsistency).
• BLEU — range: [0, 1]
  • Standard n-gram overlap metric for lexical similarity between generated and reference text.

Input / output format

Input: Molecular reference description or context, along with a question targeting a specific aspect (Structure, Property, Application, or Source).

Output: Generated molecular description or selected answer option (positive/negative).

Scoring recipe

correct = 0
total = 0
for instance in dataset:
    pred = model.generate(instance.question, instance.context)
    judgment = expert_classify(pred, instance.gold)
    if judgment == 'correct':
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Models may achieve high scores on lexical metrics (BLEU/ROUGE) while failing on factual accuracy due to hallucination or omission.
• Evaluation relies on human expert judgment for factual consistency, which is resource-intensive and may introduce annotator subjectivity.
• Scaffold-based data splitting requires careful implementation to prevent structural leakage between train and test sets.

Evidence (verbatim from paper)

We define factual accuracy as the ratio of correct predictions to the total number of slots, serving as an average metric to evaluate the reliability of the generated content.

Citation

@misc{lu2024moleculeqa,
  title={MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2403.08192}
}

• arXiv: 2403.08192