Medical LLM Merging Eval

medical-llm-merging-eval

A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs — Timilsina et al. (2025) (arXiv:2511.13373, 2025)

What this evaluates

Evaluates the effectiveness of various model merging techniques for consolidating knowledge in medical large language models. It probes whether merged models can outperform their base and parent models across diverse medical and general reasoning benchmarks.

Datasets

• MedQA — total ?; splits: test (-1)
• PubMedQA — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• MedMCQA — total ?; splits: test (-1)
• MMLU Professional Medicine — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly answered multiple-choice questions out of the total number of questions.
• standard error — range: [0, 1]
  • The standard deviation of the accuracy scores across runs or folds, used to report confidence intervals.

Input / output format

Input: Multiple-choice questions from medical and general reasoning benchmarks provided to merged LLMs.

Output: Model-generated answer choices, evaluated for correctness against gold labels.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Default pruning density for DARE-TIES (0.6) may destroy valuable medical knowledge; higher densities (0.8-0.95) are often needed.
• Merged models do not automatically outperform the base model on all tasks; broad knowledge benchmarks like MMLU Professional Medicine may see no improvement.

Evidence (verbatim from paper)

The evaluation across five medical and general reasoning benchmarks revealed significant performance differences among the merging techniques. The detailed results, including accuracy and standard error for each model, are presented in Table[IV].

Citation

@misc{timilsina2025hierarchical,
  title={A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs},
  author={Timilsina et al. (2025)},
  year={2025},
  note={arXiv:2511.13373}
}

• arXiv: 2511.13373