Molecular Embedding Benchmark Eval

molecular-embedding-benchmark-eval

Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning — Praski et al. (2025) (arXiv:2508.06199, 2025)

What this evaluates

Evaluates the quality of pretrained molecular representation learning models on downstream ADMET prediction tasks. It probes whether modern deep learning architectures (GNNs, transformers) can outperform traditional chemical fingerprints and established baselines like ECFP.

Datasets

• Collection of ADMET endpoint datasets — total ?; splits: test (-1); repo https://github.com/scikit-fingerprints/benchmarking_molecular_models

Metrics

• Mean AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve averaged across all datasets. Higher is better.
• Mean rank — range: other
  • Average rank of a model across all datasets, where rank 1 is the best performing model on that dataset. Lower is better.
• Bayesian Bradley-Terry (BBT) decision — range: other
  • Probabilistic comparison of models against a baseline (ECFP) using a hierarchical Bayesian model. Categorizes models as better, practically equivalent (within ROPE), worse, or undecidable.

Input / output format

Input: Molecular structures represented as SMILES strings, SELFIES, or molecular graphs.

Output: Predicted probabilities for ADMET endpoint classification.

Scoring recipe

auc_scores = [auroc(y_true, y_pred) for dataset in datasets]
mean_auroc = sum(auc_scores) / len(auc_scores)
ranks = [rankdata(auroc_scores_per_model)[model_idx] for dataset in datasets]
mean_rank = sum(ranks) / len(ranks)
bbt_decisions = bayesian_bradley_terry(y_true, y_pred, baseline='ECFP', rope=(0.25, 0.75), tie_threshold=0.0001)

Common pitfalls

• Mean AUROC is sensitive to outliers on unusually easy or hard datasets.
• Mean rank ignores the magnitude of performance differences between models.
• Prior evaluations often lacked rigorous statistical testing, leading to overclaiming of model superiority.

Evidence (verbatim from paper)

The aggregated results of the model, the average rank, and the AUROC, are shown in Table [1]. Aggregated metrics alone, such as mean AUROC and mean rank, are insufficient for a fair comparison of the models (Demšar [2006]; Wainer [2023]). Mean AUROC is sensitive to outliers (e.g., datasets unusually easy for certain models), while mean rank ignores the magnitude of performance differences. To address these limitations, we use the BBT model, designed specifically for robust multi-model comparisons.

Citation

@misc{praski2025benchmarking,
  title={Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning},
  author={Praski et al. (2025)},
  year={2025},
  note={arXiv:2508.06199}
}

• arXiv: 2508.06199