Mudi Eval

mudi-eval

MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions — Ngo et al. (2025) (arXiv:2506.01478, 2025)

What this evaluates

This benchmark evaluates a model's ability to predict pharmacodynamic drug-drug interactions (Synergism, Antagonism, or New Effect) using multimodal inputs including text, chemical formulas, molecular graphs, and images. It specifically probes cross-modal reasoning and generalization to unseen drug pairs under both direction-aware and direction-agnostic matching settings.

Datasets

• MUDI — total 310532; splits: test (-1); repo https://github.com/hoangbros03/MUDI

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of Precision and Recall: 2 * (P * R) / (P + R). Computed per positive class (Synergism, Antagonism, New Effect) and aggregated via Micro or Macro averaging. Negative examples (No Interaction) are excluded from metric computation.

Input / output format

Input: A pair of drugs represented by multimodal features: textual fields (name, description, SMILES, formula), molecular structure graphs, and structure images.

Output: A single categorical label: Synergism, Antagonism, or New Effect.

Scoring recipe

positive_classes = ['Synergism', 'Antagonism', 'New Effect']
f1_scores = []
for c in positive_classes:
    tp = sum(p == c and g == c for p, g in zip(preds, gold))
    fp = sum(p == c and g != c for p, g in zip(preds, gold))
    fn = sum(p != c and g == c for p, g in zip(preds, gold))
    p = tp / (tp + fp) if (tp + fp) > 0 else 0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_scores.append(2 * p * r / (p + r) if (p + r) > 0 else 0)
micro_f1 = compute_micro_f1(preds, gold, positive_classes)
macro_f1 = mean(f1_scores)

Common pitfalls

• Negative examples ('No Interaction') are used for training but must be excluded when computing evaluation metrics.
• Evaluation must explicitly state directionality: direction-aware matching treats (A, B) and (B, A) as distinct, while direction-agnostic ignores order, causing large performance gaps.
• Severe class imbalance exists for the 'New Effect' label, which drastically lowers per-class F1 scores and requires careful macro-averaging interpretation.

Evidence (verbatim from paper)

Evaluation Metrics. We report Precision, Recall, and F1 scores for each of the three positive classes, along with Micro and Macro averages. Negative examples (No Interaction) are included during training but excluded from positive-class metric computation, following common biomedical evaluation practice [21]. Evaluation Settings. We evaluate under two settings: (i) direction-aware matching, where (DRUG1, DRUG2) ≠ (DRUG2, DRUG1), and (ii) direction-agnostic matching, where order is ignored.

Citation

@misc{ngo2025mudi,
  title={MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions},
  author={Ngo et al. (2025)},
  year={2025},
  note={arXiv:2506.01478}
}

• arXiv: 2506.01478