Biological Mllm Merging Eval

biological-mllm-merging-eval

ES-Merging: Biological MLLM Merging via Embedding Space Signals — Lee et al. (2026) (arXiv:2603.14405, 2026)

What this evaluates

Evaluates the ability of merged multimodal large language models to perform cross-modal biological reasoning tasks, specifically predicting molecular interactions with proteins/cells and predicting enzyme functionality. It probes whether embedding-space-aware merging preserves modality-specific expertise better than parameter-space heuristics or fine-tuning.

Datasets

• Biological MLLM Interaction & Functionality Benchmarks — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted instances out of the total number of instances.
• Macro-F1 — range: [0, 1]
  • Unweighted mean of the F1 scores calculated for each class independently, where F1 is the harmonic mean of precision and recall.

Input / output format

Input: Multimodal inputs (molecule, protein, or cell representations) paired with task-specific few-shot in-context examples and a standardized instruction template. Unsupported modalities in baselines are converted to textual inputs.

Output: Classification predictions (e.g., interaction yes/no, inhibition/substrate specificity, or enzyme functionality classes).

Scoring recipe

def compute_metrics(predictions, gold_labels):
    acc = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = sorted(set(gold_labels))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return acc, macro_f1

Common pitfalls

• Unsupported modalities are handled differently across methods (textual conversion for baselines vs. native multimodal for ES-Merging), which can skew comparisons if not standardized.
• Task-specific fine-tuning baselines may overfit to specific datasets, leading to instability or degraded generalization on unseen cross-modal pairs.
• Evaluation relies on few-shot in-context learning, so prompt sensitivity and example selection can significantly impact reported scores.

Evidence (verbatim from paper)

We consider two instance-varying cross-modal interaction settings: molecule-protein interaction and molecule-cell interaction. For molecule-protein interaction, the task is to predict whether a given molecule interacts with a given protein, including BindingDB, BioSNAP, and Human. For molecule-cell interaction, the task is to predict the effect of a molecule on a given cell, including DrugComb and GDSC2. ... To this end, we evaluate on CYP enzyme prediction. ... We report accuracy and macro-F1 across each subset.

Citation

@misc{lee2026esmerging,
  title={ES-Merging: Biological MLLM Merging via Embedding Space Signals},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2603.14405}
}

• arXiv: 2603.14405