Drug Discovery Benchmarks Eval

drug-discovery-benchmarks-eval

MAMMAL -- Molecular Aligned Multi-Modal Architecture and Language — Shoshan et al. (2024) (arXiv:2410.22367, 2024)

What this evaluates

Evaluates a multi-modal foundation model's capability across classification, regression, and generation tasks in drug discovery. It probes the model's ability to predict cell types, assess drug efficacy and safety, design antibody CDR regions, and estimate binding affinities for proteins and small molecules.

Datasets

• Zheng68k — total 68579; splits: 5-fold CV (-1)
• MoleculeNet (BBBP/ClinTox) — total ?; splits: predefined train/val/test (-1)
• GDSC (Cancer-Drug Response 1-3) — total ?; splits: TDC random splits / 5% stratified test (-1)
• SAbDab — total ?; splits: random train/val/test with CDR3 grouping (-1)
• Weber TCR Benchmark — total 47182; splits: 10-fold CV (-1)
• SKEMPI S1131 — total 1131; splits: 10-fold CV (-1)
• DTI Benchmark — total ?; splits: unspecified (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Calculated as 2 * (precision * recall) / (precision + recall).
• Pearson — range: [-1, 1]
  • Pearson correlation coefficient measuring linear correlation between predicted and true continuous values.
• NRMSE — range: [0, ∞)
  • Normalized root mean square error. Calculated by dividing the root mean square error by the standard deviation of the labels in the test set. Lower is better.
• CDRH3-AAR — range: [0, 1]
  • CDR H3 Amino Acid Recovery rate. Measures the fraction of correctly predicted amino acids in the masked CDRH3 region.

Input / output format

Input: Textual prompts containing ranked lists of gene names (by expression level), SMILES strings for small molecules, and protein sequences (e.g., antibody FR regions or full chains). Scalar values may also be provided as part of the prompt context.

Output: Task-specific predictions: categorical labels (e.g., CL:NNNNNN cell ontology IDs or binary binding status), continuous floating-point scalars (e.g., IC50, ΔΔG, pKD), or generated amino acid sequences for masked CDR regions.

Scoring recipe

def compute_metrics(preds, golds, task_type):
    if task_type == 'cls':
        return f1_score(golds, preds, average='macro'), roc_auc_score(golds, preds)
    elif task_type == 'reg':
        return pearsonr(golds, preds)[0]
    elif task_type == 'gen':
        return amino_acid_recovery(preds, golds)
    elif task_type == 'nrmse':
        return np.sqrt(np.mean((golds - preds)**2)) / np.std(golds)

Common pitfalls

• Using different data splits than the referenced baselines (e.g., MoLFormer or TDC splits), which invalidates direct comparison.
• Treating NRMSE as a 'higher is better' metric like AUROC or F1, leading to incorrect improvement calculations.
• Skipping required preprocessing steps such as ranking genes by expression, binning values, or filtering samples with missing CDRs, which alters the input distribution.

Evidence (verbatim from paper)

We assessed model quality through a fine-tuning-based evaluation strategy, where the pretrained model is adapted to each benchmark and compared against specialized state-of-the-art (SOTA) models. ... For benchmarks with train-validation-test splits, we fine-tuned the model ibm/biomed.omics.bl.sm.ma-ted-458m on the training set, selected the best checkpoint based on validation performance, and reported final results on the test set. ... For other metrics (AUROC, CDRH3-AAR, Pearson, Spearman, and F1), higher is better.

Citation

@misc{shoshan2024mammal,
  title={MAMMAL -- Molecular Aligned Multi-Modal Architecture and Language},
  author={Shoshan et al. (2024)},
  year={2024},
  note={arXiv:2410.22367}
}

• arXiv: 2410.22367