Ming Moe Medical Eval

ming-moe-medical-eval

MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts — Liao et al. (2024) (arXiv:2404.09027, 2024)

What this evaluates

Evaluates large language models on a comprehensive suite of medical natural language processing tasks and medical licensing examinations. It probes the model's ability to process clinical text, perform information extraction, and demonstrate domain-specific knowledge and reasoning for medical exams.

Datasets

• CBLUE (via PromptCBLUE) — total ?; splits: test (-1)
• MedQA — total ?; splits: validation (-1)
• MedMCQA — total ?; splits: validation (-1)
• CMB — total ?; splits: validation (-1)
• CMExam — total ?; splits: test (-1)
• MMLU (medical subset) — total ?; splits: test (-1)
• C-Eval (medical subset) — total ?; splits: test (-1)
• CMMLU (medical subset) — total ?; splits: test (-1)
• 2023 Chinese National Pharmacist Licensure Examination — total ?; splits: test (-1)

Metrics

• score (primary) — range: percent
  • Percentage of correctly predicted options or task-specific scores (e.g., F1, Accuracy) averaged across tasks. For multiple-choice exams, exact match between the model's selected option and the ground truth.

Input / output format

Input: Multiple-choice medical questions or clinical text prompts formatted for LLM consumption (e.g., via PromptCBLUE for CBLUE tasks).

Output: Model generates a textual response, typically selecting a letter option (A, B, C, D) or providing a direct answer. For CBLUE tasks, task-specific structured outputs.

Scoring recipe

def compute_score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_opt = normalize_option(pred)
        if pred_opt == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• MedQA, MedMCQA, and CMB are evaluated on their validation sets according to the paper, not the standard test sets.
• For general benchmarks (MMLU, C-Eval, CMMLU), only medical-domain questions are extracted and evaluated, not the full benchmark.
• CBLUE tasks use PromptCBLUE to convert structured tasks into pure text prompts, which may affect baseline comparability if not applied consistently.

Evidence (verbatim from paper)

The examination benchmarks include: the US test set of MedQA(Jin et al., 2021), the development set of MedMCQA(Pal et al., 2022), and two comprehensive Chinese medical exam datasets, CMB(Wang et al., 2023c) and CMExam(Liu et al., 2024a). We also collect the medical parts of the general benchmarks, which include: MMLU(Hendrycks et al., 2020), C-Eval(Huang et al., 2023), and CMMLU(Li et al., 2023a). Evaluation was done using validation data for MedQA, MedMCQA, and CMB.

Citation

@misc{liao2024mingmoe,
  title={MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts},
  author={Liao et al. (2024)},
  year={2024},
  note={arXiv:2404.09027}
}

• arXiv: 2404.09027