qhjqhj00 molecule-net-eval
SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision — Zheng et al. (2024) (arXiv:2412.05569, 2024)
What this evaluates
Evaluates a model's ability to predict molecular properties from SMILES strings by fine-tuning on 7 classification benchmarks and measuring performance under scaffold splitting.
Datasets
- MoleculeNet — total ?; splits: test (-1)
Metrics
ROC-AUC(primary) — range: [0, 1]- Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
Input / output format
Input: Normalized SMILES strings representing molecular structures.
Output: Predicted class probabilities or binary/multi-class labels for each molecular property task.
Scoring recipe
def compute_roc_auc(y_true, y_pred_proba):
fpr, tpr, _ = roc_curve(y_true, y_pred_proba)
auc_score = auc(fpr, tpr)
return auc_score
Common pitfalls
- Uses scaffold splitting rather than random splitting, which is stricter and tests generalization to unseen chemical scaffolds.
- Evaluates each of the 7 tasks separately rather than aggregating scores across tasks before computing the mean.
- Requires normalized SMILES inputs, which must be generated using a specific regular expression tokenizer.
Evidence (verbatim from paper)
We evaluate SMI-EDITOR on the MoleculeNet (Wu et al., 2017) benchmark and compare its performance with baseline models. We evaluate SMI-EDITOR on 7 widely-used molecular property prediction tasks ( see Appendix H for details). For all the seven tasks, we take the normalized SMILES information as model input and fine-tuning on each task separately. We use ROC-AUC as the evaluation metric, and the results are summarized in Table 1.
Citation
@misc{zheng2024smieditor,
title={SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision},
author={Zheng et al. (2024)},
year={2024},
note={arXiv:2412.05569}
}
- arXiv: 2412.05569