Molvision Benchmark Eval

molvision-benchmark-eval

MolVision: Molecular Property Prediction with Vision Language Models — Adak et al. (2025) (arXiv:2507.03283, 2025)

What this evaluates

Evaluates vision-language models on molecular property prediction by combining skeletal structure images with textual prompts. It probes the model's ability to perform binary classification, numerical regression, and textual description generation across diverse chemical properties.

Datasets

• BACE-V — total ?; splits: train (1210), test (303)
• BBBP-V — total ?; splits: train (1640), test (410)
• HIV-V — total ?; splits: train (32902), test (8225)
• ClinTox-V — total ?; splits: train (1193), test (298)
• Tox21-V — total ?; splits: train (6265), test (1566)
• ESOL-V — total ?; splits: train (902), test (226)
• LD50-V — total ?; splits: train (5908), test (1477)
• QM9-V — total ?; splits: train (107000), test (27000)
• PCQM4Mv2-V — total ?; splits: train (3000000), test (700000)
• ChEBI-V — total ?; splits: train (32000), test (8000)

Metrics

• True/False accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted binary labels (True/False) out of total test instances. Computed as correct predictions divided by total predictions.
• RMSE — range: other
  • Root Mean Squared Error between predicted and ground-truth numerical values for regression tasks.
• Text similarity — range: [0, 1]
  • Semantic or lexical overlap score between generated molecular descriptions and reference text.

Input / output format

Input: Molecular skeletal structure image paired with a manually engineered text prompt containing SMILES or SELFIES representation.

Output: Textual response: 'True' or 'False' for classification tasks; a numerical value for regression tasks; or a descriptive sentence for description tasks.

Scoring recipe

def compute_metric(predictions, golds, task_type):
    if task_type == 'classification':
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.lower())
        return correct / len(predictions)
    elif task_type == 'regression':
        errors = [(float(p) - float(g))**2 for p, g in zip(predictions, golds)]
        return (sum(errors) / len(errors)) ** 0.5
    else:
        return text_similarity_score(predictions, golds)

Common pitfalls

• Visual data alone is insufficient for accurate prediction; multimodal fusion is required for robust performance.
• Datasets use RDKit-generated skeletal images rather than experimental crystal structures, which may introduce representation bias.
• Manual prompt engineering limits direct comparison across different VLM architectures without standardized prompting.

Evidence (verbatim from paper)

For classification, we frame the task as a True/False question, where the model predicts whether a molecule inhibits a target property. For regression, the model generates a numerical value representing the target property and for description task, the model generates textual output.

Citation

@misc{adak2025molvision,
  title={MolVision: Molecular Property Prediction with Vision Language Models},
  author={Adak et al. (2025)},
  year={2025},
  note={arXiv:2507.03283}
}

• arXiv: 2507.03283