Medeval Eval

medeval-eval

MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation — He et al. (2023) (arXiv:2310.14088, 2023)

What this evaluates

Evaluates language models on multi-level (sentence/document) and multi-task (NLU/NLG) medical benchmarks across diverse clinical domains. It probes a model's ability to perform clinical text classification, report code prediction, and medical report summarization using both fine-tuned PLMs and prompted LLMs.

Datasets

• MedEval — total ?; splits: (unstated); repo https://github.com/ZexueHe/MedEval

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels or report codes out of the total number of test instances.
• exact match rate (EMR) — range: [0, 1]
  • Proportion of predictions that exactly match the ground-truth label or code string.
• ROUGE-1/2/L — range: [0, 1]
  • Standard ROUGE recall/precision/f1 scores measuring n-gram overlap between generated and reference clinical summaries.
• BLEU4 — range: [0, 1]
  • Geometric mean of modified n-gram precisions up to 4-grams, penalizing brevity in generated text.

Input / output format

Input: Clinical text instances (individual sentences or full EMR documents) provided as prompts. For LLM evaluation, prompts include zero-shot or few-shot examples with explicit instructions for classification or generation tasks.

Output: For NLU tasks: discrete class labels (e.g., abnormality/ambiguity flags or report codes). For NLG tasks: free-text generated sentences or summarized clinical reports.

Scoring recipe

def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    emr = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    rouge_scores = rouge_score(golds, predictions) # ROUGE-1, 2, L
    bleu4 = bleu_score(golds, predictions)
    return {'accuracy': acc, 'emr': emr, 'rouge': rouge_scores, 'bleu4': bleu4}

Common pitfalls

• Models lacking instruction tuning (e.g., BioMed LM) fail to follow prompts or hit token limits, yielding artificially low scores.
• Few-shot prompting consistently improves NLU accuracy but shows no reliable trend for NLG generation tasks.
• Performance heavily depends on domain familiarity; models excel on common domains (e.g., chest) but struggle on rare/ambiguous ones (miscellaneous).

Evidence (verbatim from paper)

We report the average accuracy over all classes of diseases and the exact match rate (EMR) between predictions and labels. ... We report the Rouge scores and BLEU4 scores.

Citation

@misc{he2023medeval,
  title={MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2310.14088}
}

• arXiv: 2310.14088