Madrigal Drug Comb Eval

madrigal-drug-comb-eval

Multimodal AI predicts clinical outcomes of drug combinations from preclinical data — Huang et al. (2025) (arXiv:2503.02781, 2025)

What this evaluates

Evaluates a multimodal AI model's ability to predict clinical outcomes and adverse reactions for drug combinations from preclinical data. It probes robustness to missing modalities and generalization to novel drugs under strict hold-out splits.

Datasets

• TWOSIDES — total 4656138; splits: split-by-drug pairs (-1), split-by-drugs (random) (-1), split-by-drugs (target) (-1), split-by-drugs (ATC) (-1)
• DrugBank — total 1188371; splits: split-by-drug pairs (-1), split-by-drugs (random) (-1), split-by-drugs (target) (-1), split-by-drugs (ATC) (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
• AUPRC — range: [0, 1]
  • Area under the precision-recall curve. Measures the trade-off between precision and recall across all classification thresholds, particularly sensitive to class imbalance.
• Fmax — range: [0, 1]
  • Maximum F-measure across all classification thresholds. The highest value of the harmonic mean of precision and recall.

Input / output format

Input: Pair of drugs encoded via modality-specific encoders (chemical structure, molecular pathway knowledge graph, transcriptomic responses, cell viability data). Embeddings are fused via an attention bottleneck to produce a single multimodal embedding per drug pair.

Output: A continuous prediction score for each of the 953 clinical outcomes/adverse reactions.

Scoring recipe

def compute_metrics(predictions, labels, thresholds=np.linspace(0, 1, 1000)):
    preds_binary = (predictions[:, None] >= thresholds[None, :]).astype(int)
    tpr = preds_binary.sum(axis=0) / labels.sum(axis=0)
    fpr = preds_binary.sum(axis=0) / (1 - labels).sum(axis=0)
    auroc = np.trapz(tpr, fpr)
    precision = preds_binary.sum(axis=0) / preds_binary.sum(axis=0).clip(min=1)
    recall = preds_binary.sum(axis=0) / labels.sum(axis=0)
    f1 = 2 * precision * recall / (precision + recall).clip(min=1e-9)
    auprc = np.trapz(precision, recall)
    fmax = f1.max()
    return {'AUROC': auroc, 'AUPRC': auprc, 'Fmax': fmax}

Common pitfalls

• Data leakage if splits are not strictly by-drug rather than by-drug-pair or random.
• Test-time modality asymmetry: Madrigal is restricted to preclinically available modalities while baselines receive full inputs, making direct comparison harder.
• Multi-task nature: 953 outcomes are predicted simultaneously; performance varies significantly by outcome type and biological pathway specificity.

Evidence (verbatim from paper)

We evaluate Madrigal on two settings: (1) holding out all samples for specific drug pairs (“split-by-drug pairs”) and (2) holding out all samples for specific drugs together with any of their pairings (“split-by-drugs”). The split-by-drugs setting better reflects prediction for a novel compound combined with an approved partner (Fig.[2]a). ... We report area under receiver-operator curve (AUROC), area under precision-recall curve (AUPRC), and maximum F measure (Fmax) (Methods Sec.[3.3]).

Citation

@misc{huang2025madrigal,
  title={Multimodal AI predicts clinical outcomes of drug combinations from preclinical data},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2503.02781}
}

• arXiv: 2503.02781