Diabetica Eval

diabetica-eval

Diabetica: Adapting Large Language Model to Enhance Multiple Medical Tasks in Diabetes Care and Management — Lai Wei et al. (2024) (arXiv:2409.13191, 2024)

What this evaluates

Evaluates large language models on diabetes care and management tasks, probing their ability to recall foundational medical knowledge, make clinical decisions in case studies, generate precise text, and reason through open-ended patient queries.

Datasets

• Diabetica — total 467; splits: multiple_choice (312), fill_in_blank (35), open_ended (120)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered multiple-choice questions out of the total number of questions.
• BERTScore — range: [0, 1]
  • Similarity metric based on contextual embeddings from BERT, comparing generated text against reference answers.
• ROUGE-L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation based on longest common subsequence.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap between generated and reference text.
• ROUGE-2 — range: [0, 1]
  • Bigram overlap between generated and reference text.
• BLEU — range: [0, 1]
  • Bilingual Evaluation Understudy measuring n-gram precision with brevity penalty.
• LLM-judge score — range: [1, 10]
  • Rating on a 1-10 scale assigned by GPT-4 or Claude-3.5 acting as judges, based on physician-annotated evaluation rules.

Input / output format

Input: Multiple-choice: question stem with options. Fill-in-the-blank: sentence with a masked token. Open-ended: category label, clinical question, and physician-annotated evaluation rules.

Output: Multiple-choice: selected option letter or text. Fill-in-the-blank: the missing word or phrase. Open-ended: a free-form clinical response or dialogue turn.

Scoring recipe

# For multiple-choice:
accuracy = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)

# For fill-in-the-blank:
bertscore = compute_bertscore(predictions, golds)
rouge_l = compute_rouge(predictions, golds, rouge_type='L')
rouge_1 = compute_rouge(predictions, golds, rouge_type='1')
rouge_2 = compute_rouge(predictions, golds, rouge_type='2')
bleu = compute_bleu(predictions, golds)

# For open-ended:
judge_scores = []
for pred, rules in zip(predictions, rules_list):
    prompt = f"Rate this answer (1-10) based on these rules: {rules}\nAnswer: {pred}"
    score = llm_judge(prompt) # GPT-4 or Claude-3.5
    judge_scores.append(score)
avg_judge_score = sum(judge_scores) / len(judge_scores)

Common pitfalls

• ROUGE/BLEU metrics penalize semantically correct but lexically different medical terms or phrasing.
• LLM-as-a-judge scores may be biased by response verbosity, formatting, or the specific judge model used (GPT-4 vs Claude-3.5).
• Zero-shot prompting for baselines may not fairly compare against fine-tuned models if instruction formats differ.

Evidence (verbatim from paper)

We used accuracy that measures the percentage of correct answers given by a model for multiple-choice questions. ... We employed strong LLMs (GPT-4 and Claude-3.5) as judges to evaluate these models on open-ended questions and rate each answer on a scale of 1-10 based on the human rule.

Citation

@misc{wei2024diabetica,
  title={Diabetica: Adapting Large Language Model to Enhance Multiple Medical Tasks in Diabetes Care and Management},
  author={Lai Wei et al. (2024)},
  year={2024},
  note={arXiv:2409.13191}
}

• arXiv: 2409.13191