Bioinstruct Eval

bioinstruct-eval

BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing — Tran et al. (2023) (arXiv:2310.19975, 2023)

What this evaluates

Evaluates large language models on biomedical natural language processing tasks, including multiple-choice question answering, natural language inference, clinical information extraction, and text generation. It probes the model's ability to follow domain-specific instructions, extract precise medical entities, and generate coherent clinical notes or answers.

Datasets

• MedQA-USMLE — total ?; splits: test (-1)
• MedMCQA — total ?; splits: test (-1)
• PubmedQA — total ?; splits: test (-1)
• BioASQ MCQA — total ?; splits: test (-1)
• MedNLI — total ?; splits: test (-1)
• Medication Status Extraction — total ?; splits: test (-1)
• Coreference Resolution — total ?; splits: test (-1)
• Conv2note — total ?; splits: test (-1)
• ICliniq — total ?; splits: test (-1)
• MediQA-Task A — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted multiple-choice answers or NLI labels.
• Precision/Recall/F1 — range: [0, 1]
  • Standard information extraction metrics computed over extracted clinical entities and their attributes.
• Conditional ACC — range: [0, 1]
  • Accuracy of medication status classification computed only on instances where the target medication was correctly extracted.
• GPT-4 Coherence/Completeness/Naturalness — range: [1, 5]
  • LLM-as-a-judge scores (typically 1-5 scale) generated by prompting GPT-4 to evaluate the quality of generated clinical notes or answers.
• Concept_F1 — range: [0, 1]
  • F1 score computed over biomedical concept matches between generated text and reference.
• BertScore_F1 — range: [0, 1]
  • F1 score based on contextual embeddings from BERT, measuring semantic similarity between generated and reference text.
• Bleurt — range: [0, 1]
  • Learned evaluation metric using BERT-based representations to score semantic similarity.

Input / output format

Input: Instruction prompt containing a biomedical question, clinical context, or task description (e.g., doctor-patient dialogue, medical text).

Output: Model-generated text response, typically a multiple-choice letter, short answer, extracted entities, or a clinical note.

Scoring recipe

def score(predictions, gold, task_type):
    if task_type in ['QA', 'NLI']:
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'IE':
        # Compute Precision, Recall, F1 on extracted entities
        # Compute Conditional ACC: accuracy of status classification given correct extraction
        pass
    elif task_type == 'Generation':
        # GPT-4 scores (1-5) for Coherence, Completeness, Naturalness via API
        # Concept_F1, BertScore_F1, Bleurt via standard libraries
        pass
    return metric_value

Common pitfalls

• GPT-4 evaluation metrics (Coherence, Completeness, Naturalness) are subjective and highly sensitive to the prompt template, temperature, and model version used for scoring.
• Conditional Accuracy for medication status extraction penalizes models for incorrect entity extraction before evaluating classification, conflating two distinct sub-tasks and potentially masking classification capability.
• Multi-task instruction tuning does not uniformly improve all task types; IE tasks often plateau or degrade when combined with generative tasks, contrary to the expectation that more tasks always help.

Evidence (verbatim from paper)

Table 1 illustrates results from several QA benchmarks in the biomedical domain, including MedQA-USMLE, MedMCQA, PubmedQA, and BioASQ MCQA. In the MedQA-USMLE, while the base LLaMA 1 7B secured 27.1, its instructed version (LLaMA 1 7B Instruct) rose to 31.58. This positive trend extends to tasks such as MedMCQA... Table 2 showcases the performance of various models on the clinical information extraction task, specifically focused on medication status... Table 3 encapsulates the performances of various models on the Conv2note, a challenging task aimed at converting doctor-patient conversations to clinical notes. Evidently, instruction tuning models, such as LLaMA 1 7B Instruct, consistently surpass their original versions, particularly in GPT-4 measures like Coherence, Completeness, and Naturalness. Specifically, LLaMA 1 7B Instruct sees a marked enhancement in Coherence, moving from 2.67 to 4.49. Furthermore, the instruction tuning also exhibits its strength in grasping medical concepts with LLaMA 7B Instruct attaining a Concept_F1 score of 31.51, a vast improvement over LLaMA 1 7B's 16.9.

Citation

@misc{tran2023bioinstruct,
  title={BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing},
  author={Tran et al. (2023)},
  year={2023},
  note={arXiv:2310.19975}
}

• arXiv: 2310.19975