Entropy Adaptive Merging Eval

entropy-adaptive-merging-eval

The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging — Ambekar et al. (2026) (arXiv:2602.21372, 2026)

What this evaluates

Evaluates the robustness of an entropy-adaptive model merging framework under heterogeneous domain shifts. It probes whether a merged model can adapt to unseen target domains using only forward passes and entropy-based coefficients, without backpropagation or labeled target data.

Datasets

• MiDog Atypical — total 454; splits: test (-1)
• Organs — total 1645; splits: test (-1)
• Histopantum — total 281142; splits: test (-1)
• ISIC Skin — total 10015; splits: test (-1)
• Messidor — total 1200; splits: test (-1)
• PACS — total 9991; splits: test (-1)
• VLCS — total 10729; splits: test (-1)
• Office-Home — total 15500; splits: test (-1)
• TerraIncognita — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified samples over the target stream or batch. Calculated as (number of correct predictions) / (total samples).

Input / output format

Input: A batch of target-domain images (typically size 32) fed into K frozen, pre-trained domain-specific ViT models.

Output: Predicted class labels for each image in the batch.

Scoring recipe

correct = 0
total = 0
for batch in target_stream:
    preds = merged_model(batch)
    correct += (preds == batch.labels).sum()
    total += len(batch)
accuracy = (correct / total) * 100

Common pitfalls

• Assuming backpropagation or target-time updates are used at test time; the protocol explicitly requires a forward-only pass with no backpropagation.
• Ignoring the leave-one-domain-out domain generalization setup; each model is trained on a single source domain and evaluated on a completely held-out target domain.
• Applying naive mean merging without decoupling encoder and classifier weights, which the paper shows causes severe performance degradation due to layer-wise misalignment.

Evidence (verbatim from paper)

We report accuracy over the target stream. All experiments are run on NVIDIA A100 GPUs; our method is non-parametric and incurs only lightweight per-batch overhead. Table[1] reports medical-domain results for both ViT-B/32 and ViT-B/16 under the standard leave-one-domain-out domain generalization protocol, using both offline and online merging with a test-time batch size of 32 samples.

Citation

@misc{ambekar2026mean,
  title={The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging},
  author={Ambekar et al. (2026)},
  year={2026},
  note={arXiv:2602.21372}
}

• arXiv: 2602.21372