Protein Ligand Binding Eval

protein-ligand-binding-eval

Hermes: Large DEL Datasets Train Generalizable Protein-Ligand Binding Prediction Models — Kleinsasser et al. (2026) (arXiv:2602.13503, 2026)

What this evaluates

Evaluates a model's ability to predict protein-ligand binding using only sequence data. It probes generalization across diverse protein targets, novel chemical scaffolds, and external benchmarks by measuring ranking performance between binders and decoys.

Datasets

• DEL Protein Split — total ?; splits: test (-1)
• DEL Chemical Library Split — total ?; splits: test (-1)
• MF-PCBA — total ?; splits: test (-1)
• Public Binders/Decoys — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen binder is ranked higher than a randomly chosen non-binder/decoy.

Input / output format

Input: Protein amino acid sequence and ligand molecular representation (SMILES/sequence), processed via pre-trained embeddings (ESM2 and ChemBERTa) with cross-attention.

Output: Binary binding prediction or continuous binding probability score.

Scoring recipe

auroc_scores = []
for target in targets:
    preds = get_predictions(target)
    labels = get_labels(target)
    auroc_scores.append(roc_auc_score(labels, preds))
mean_auroc = np.mean(auroc_scores)
std_auroc = np.std(auroc_scores)

Common pitfalls

• N in evaluation tables refers to the number of protein targets, not the number of ligand pairs.
• Some benchmarks (e.g., MF-PCBA) may contain data overlapping with competitor models' training sets, inflating apparent performance.
• Datasets are often subsampled (e.g., to 50k samples) for inference feasibility, which may not reflect full-dataset evaluation conditions.

Evidence (verbatim from paper)

Table 3: Hermes vs benchmarks per-protein AUROC comparison. AUROC columns indicate mean + standard deviation AUROC scores across all targets in the evaluation set.

Citation

@misc{kleinsasser2026hermes,
  title={Hermes: Large DEL Datasets Train Generalizable Protein-Ligand Binding Prediction Models},
  author={Kleinsasser et al. (2026)},
  year={2026},
  note={arXiv:2602.13503}
}

• arXiv: 2602.13503