Biomed Enriched Eval

biomed-enriched-eval

Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content — Touchent et al. (2025) (arXiv:2506.20331, 2025)

What this evaluates

Evaluates the biomedical knowledge, clinical reasoning, and domain-specific comprehension of language models using multiple-choice question-answering benchmarks across anatomy, clinical medicine, genetics, and multilingual medical QA.

Datasets

• MMLU Professional Medicine — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• MedMCQA — total ?; splits: test (-1)
• PubMedQA — total ?; splits: test (-1)
• FrenchMedMCQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: Multiple-choice questions covering biomedical, clinical, and biological topics, typically presented with a question stem and four or five options.

Output: The model's predicted answer choice (letter or full text) for each question.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
return (correct / len(gold_labels)) * 100

Common pitfalls

• Assuming 'Clinical Knowledge' and 'Medical Genetics' are separate datasets rather than MMLU subsets or specialized benchmarks without checking the paper's appendix.
• Comparing data efficiency claims without normalizing for the exact number of training tokens, as the paper highlights performance gains using only 1/3 of the baseline tokens.
• Overlooking that FrenchMedMCQA results are reported in accuracy but may use a different option format or language-specific preprocessing than the English benchmarks.

Evidence (verbatim from paper)

BE-French showed clear improvements in French medical QA (FrenchMedMCQA), achieving 40.5% accuracy, significantly surpassing BE-Base and the OLMo2-7B-stage1 baseline (38.32%, Figure 1).

Citation

@misc{touchent2025biomedenriched,
  title={Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content},
  author={Touchent et al. (2025)},
  year={2025},
  note={arXiv:2506.20331}
}

• arXiv: 2506.20331