qhjqhj00 tox21-fsl-eval
Benchmarking Toxic Molecule Classification using Graph Neural Networks and Few Shot Learning — Mehta et al. (2023) (arXiv:2311.13490, 2023)
What this evaluates
Evaluates graph neural networks for few-shot toxic molecule classification. It probes the model's ability to generalize from very limited labeled examples (shots) and adapt to new query sets using meta-learning and graph augmentation techniques.
Datasets
- Tox21 — total ?; splits: train (-1), val (-1)
Metrics
ROC-AUC Score(primary) — range: [0, 1]- Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
Validation Accuracy— range: percent- Percentage of correctly classified instances in the validation set.
Input / output format
Input: Molecular graphs represented as node and edge feature matrices, processed under a few-shot learning protocol with 10 training shots and 15 query examples per task.
Output: Predicted toxicity class label (or probability scores) for each molecule in the query set.
Scoring recipe
def compute_metrics(preds, labels):
acc = np.mean(preds == labels) * 100
fpr, tpr, _ = roc_curve(labels, preds[:, 1])
auc = auc(fpr, tpr)
return acc, auc
Common pitfalls
- Few-shot splits are highly sensitive to the specific selection of the 10-shot training set and 15-query examples; results may vary significantly across different random seeds.
- GNN performance is heavily influenced by node ordering and graph isomorphism; models like GCN are sensitive to permutation while GIN is not, affecting reproducibility if graph canonicalization is skipped.
- Class imbalance in toxic molecule datasets can skew accuracy; ROC-AUC is a more robust metric but requires proper probability calibration.
Evidence (verbatim from paper)
The baseline model, GCN, achieves a validation accuracy of 65.02% and a ROC-AUC score of 0.732. Comparatively, the GCN+FLAG model showcases an improvement with a validation accuracy of 70.68% (+5.66%) and a ROC-AUC score of 0.806 (+0.074). The GIN model demonstrates the highest validation accuracy of 73.23% (+8.21%) and a ROC-AUC score of 0.816 (+0.084).
Citation
@misc{mehta2023benchmarking,
title={Benchmarking Toxic Molecule Classification using Graph Neural Networks and Few Shot Learning},
author={Mehta et al. (2023)},
year={2023},
note={arXiv:2311.13490}
}
- arXiv: 2311.13490