Imdrug Eval

imdrug-eval

ImDrug: A Benchmark for Deep Imbalanced Learning in AI-aided Drug Discovery — Lanqing Li et al. (2022) (arXiv:2209.07921, 2022)

What this evaluates

Evaluates deep learning models for imbalanced and long-tailed classification and regression in AI-aided drug discovery. It probes model robustness to severe class imbalance, open long-tailed distributions, and out-of-distribution chemical splits across graph, sequence, and fingerprint molecular representations.

Datasets

• HIV — total 41127; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug
• SBAP — total 32140; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug
• USPTO-50K — total 50036; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug
• DrugBank — total 191808; splits: random (-1), standard (-1); repo https://github.com/DrugLT/ImDrug

Metrics

• Balanced-Acc (primary) — range: [0, 1]
  • Macro-average of per-class recall: mean over all classes of (true positives / total actual positives) for each class. Insensitive to label distribution.
• Balanced-F1 — range: [0, 1]
  • Macro-average of per-class F1 scores: harmonic mean of per-class precision and recall, averaged across all classes.
• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.

Input / output format

Input: Molecular structures encoded as graphs (via DGL), SMILES sequences, or Morgan fingerprints. Tasks support single-instance or multi-instance (bag-level) prediction.

Output: Predicted class labels or probabilities for classification tasks; continuous values for regression tasks.

Scoring recipe

def balanced_accuracy(y_true, y_pred, num_classes):
    recalls = []
    for c in range(num_classes):
        mask = y_true == c
        if mask.sum() > 0:
            recalls.append((y_pred[mask] == y_true[mask]).mean())
        else:
            recalls.append(0.0)
    return np.mean(recalls)

def balanced_f1(y_true, y_pred, num_classes):
    f1s = []
    for c in range(num_classes):
        mask = y_true == c
        if mask.sum() > 0:
            tp = (y_pred[mask] == y_true[mask]).sum()
            fp = (y_pred[mask] != y_true[mask]).sum()
            fn = (y_pred[y_true == c] != y_true[mask]).sum()
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
            f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    return np.mean(f1s)

Common pitfalls

• Relying solely on AUROC, which the paper notes is over-optimistic and insensitive to baseline differences in highly imbalanced drug discovery settings.
• Using standard random splits instead of the proposed chemically realistic 'standard' splits, which masks real-world out-of-distribution generalization challenges.
• Ignoring multi-instance prediction settings (e.g., SBAP, DrugBank) where bag-level labels require specialized evaluation rather than instance-level averaging.

Evidence (verbatim from paper)

In experiments, we benchmark 11 baselines for conventional imbalanced & long-tailed classification (Sec.[3.3.2]), as well as 5 additional baselines tailored for open LT and imbalanced regression, by reporting the 2 proposed balanced accuracy and balanced F1 measures along with the conventional AUROC. Other metrics for imbalanced learning such as AUPRC and Weighted-F1 are also reported in Appendix H. For hybrid prediction tasks/datasets (Table[II]), unless otherwise specified, the training mode is single-instance prediction by default. All average performance with standard deviations is evaluated over 3 random seeds.

Citation

@misc{li2022imdrug,
  title={ImDrug: A Benchmark for Deep Imbalanced Learning in AI-aided Drug Discovery},
  author={Lanqing Li et al. (2022)},
  year={2022},
  note={arXiv:2209.07921}
}

• arXiv: 2209.07921