Lo Hi Eval

lo-hi-eval

Lo-Hi: Practical ML Drug Discovery Benchmark — Steshin et al. (2023) (arXiv:2310.06399, 2023)

What this evaluates

Evaluates machine learning models on two drug discovery tasks: Hit Identification (predicting activity for novel, structurally dissimilar molecules) and Lead Optimization (ranking minor molecular modifications to predict activity changes). It probes a model's ability to generalize to unseen chemical space and capture fine-grained structure-activity relationships.

Datasets

• DRD2-Hi — total 6000; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023
• HIV-Hi — total 40000; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023
• KDR-Hi — total ?; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023
• Sol-Hi — total ?; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023
• DRD2-Lo — total ?; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023
• KCNH2-Lo — total ?; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023
• KDR-Lo — total ?; splits: train (-1), test (-1); repo https://github.com/SteshinSS/lohi_neurips2023

Metrics

• PR AUC (primary) — range: [0, 1]
  • Precision-Recall Area Under the Curve. Computed as the area under the precision-recall curve for binary classification. Normalized to [0, 1]. Favors models that rank positive instances early.
• Spearman's correlation coefficient (primary) — range: [-1, 1]
  • Rank-based correlation calculated within each molecule cluster and averaged across all clusters. Ranges from -1 (perfectly inverse ranking) to 1 (perfect ranking), with 0 indicating random performance.

Input / output format

Input: Molecular structures provided as graphs (for GNNs like Chemprop) or binary fingerprints (ECFP4 or MACCS keys) for traditional ML models.

Output: For Hi: binary probability or label (active/inactive). For Lo: continuous activity score or ranking score for each molecule within a cluster.

Scoring recipe

def score_hi(y_true, y_pred):
    prec, rec, _ = precision_recall_curve(y_true, y_pred)
    return auc(rec, prec)

def score_lo(y_true_cluster, y_pred_cluster):
    return spearmanr(y_true_cluster, y_pred_cluster).correlation

def score_lo_overall(all_true_clusters, all_pred_clusters):
    return mean(score_lo(t, p) for t, p in zip(all_true_clusters, all_pred_clusters))

Common pitfalls

• Using standard random or scaffold splits instead of the strict Hi-splitter (Tanimoto similarity < 0.4 between train and test), which artificially inflates generalization performance.
• Evaluating the Lo task with accuracy or RMSE instead of within-cluster Spearman correlation, as the task specifically requires ranking minor modifications rather than absolute activity prediction.
• Averaging results across random seeds instead of across the three distinct data folds, as the benchmark protocol mandates hyperparameter tuning on fold 1 and evaluation on folds 2 and 3.

Evidence (verbatim from paper)

For our benchmark, we have selected the PR AUC. As a simple binary classification metric without parameters, it is implemented in most libraries and normalized to a range of [0, 1]. The PR AUC favors early recognition models and does not appeal to wrong intuition among readers in an unbalanced setting. Our goal is to determine whether the models can make better predictions than assuming "the modified molecule active in the same manner as the original hit." We chose Spearman’s correlation coefficient as our metric, calculated within each cluster and averaged across clusters.

Citation

@misc{steshin2023lohi,
  title={Lo-Hi: Practical ML Drug Discovery Benchmark},
  author={Steshin et al. (2023)},
  year={2023},
  note={arXiv:2310.06399}
}

• arXiv: 2310.06399