Medical Benchmarks Eval

medical-benchmarks-eval

Aloe: A Family of Fine-tuned Open Healthcare LLMs — Gururajan et al. (2024) (arXiv:2405.01886, 2024)

What this evaluates

Evaluates large language models on medical knowledge, clinical reasoning, and safety alignment using multiple-choice and open-ended healthcare QA tasks. It measures standard accuracy across major medical benchmarks and quantifies unsafe response rates via automated safety classifiers.

Datasets

• MultiMedQA — total ?; splits: test (-1)
• MedMCQA — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• PubMedQA — total ?; splits: test (-1)
• MMLU Med. — total ?; splits: test (-1)
• CareQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted options out of total questions. For MultiMedQA, a weighted average accuracy across subtasks is computed. An overall arithmetic mean is calculated across all medical datasets.
• Attack Success Rate (ASR) — range: [0, 1]
  • Ratio of unsafe answers over total evaluated responses, classified using Llama Guard 2.

Input / output format

Input: Medical multiple-choice questions, optionally augmented with 5 nearest-neighbor few-shot examples (Medprompt) or evaluated via self-consistency Chain-of-Thought (5 or 20 ensembles with shuffled options).

Output: Step-by-step reasoning followed by a selected option (A, B, C, etc.), or direct option selection. Final answer determined via majority voting across ensemble runs.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_medical_avg(results):
    non_careqa = {k: v for k, v in results.items() if k != 'CareQA'}
    weighted_multi = sum(v * w for v, w in non_careqa.items())
    overall = sum(results.values()) / len(results)
    return weighted_multi, overall

Common pitfalls

• Ensemble size (5 vs 20) in Medprompt/SC-CoT yields only ~1% accuracy gain but increases compute cost by 4x.
• Medprompt few-shot examples must be drawn from the benchmark's training set (or MedMCQA's if unavailable), limited to 20k random samples to save compute.
• ASR scores are highly sensitive to the underlying safety classifier (Llama Guard 2) and jailbreak attack styles.

Evidence (verbatim from paper)

We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets.

Citation

@misc{gururajan2024aloe,
  title={Aloe: A Family of Fine-tuned Open Healthcare LLMs},
  author={Gururajan et al. (2024)},
  year={2024},
  note={arXiv:2405.01886}
}

• arXiv: 2405.01886