Multimedbench Eval

multimedbench-eval

Towards Generalist Biomedical AI — Tu et al. (2023) (arXiv:2307.14334, 2023)

What this evaluates

Evaluates a generalist biomedical AI model's ability to process multimodal clinical data across diverse tasks. It probes in-distribution performance on standard biomedical benchmarks, zero-shot generalization to unseen medical concepts like tuberculosis, and the clinical applicability of generated radiology reports.

Datasets

• MultiMedBench — total ?; splits: test (-1)
• Montgomery County Chest X-ray — total 138; splits: test (138)
• MIMIC-CXR — total ?; splits: test (246)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct yes/no predictions for the tuberculosis detection task. Calculated as the number of correct predictions divided by the total number of test cases.
• task-specific metrics — range: other
  • Standard evaluation metrics appropriate for each individual task in the MultiMedBench mixture, computed on the test split under the same few-shot setup used during training.
• radiologist preference rate — range: percent
  • Percentage of cases where a qualified radiologist ranks a model-generated report finding as superior to alternatives in a side-by-side evaluation, or the frequency of clinically significant errors/omissions in independent evaluation.

Input / output format

Input: Multimodal inputs comprising clinical text (e.g., indication sections, prompts) and medical images (e.g., frontal chest X-rays). For zero-shot reasoning tasks, text-only prompts with dummy image placeholders are used.

Output: Text outputs including yes/no answers for classification tasks, structured radiology report findings, and class predictions with accompanying descriptive reports.

Scoring recipe

def score_classification(predictions, gold_labels):
    return sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)

def score_reports(radiologists, model_reports, reference_reports, images):
    errors = []
    omissions = []
    for report, ref, img in zip(model_reports, reference_reports, images):
        e, o = radiologists.annotate_disagreements(report, ref, img)
        errors.extend(e)
        omissions.extend(o)
    return {'errors': errors, 'omissions': omissions, 'significance': radiologists.assess_clinical_impact(errors + omissions)}

Common pitfalls

• Assuming automatic NLG metrics are sufficient for evaluating clinical report quality; the paper explicitly requires expert radiologist evaluation for clinical applicability.
• Confusing zero-shot with one-shot evaluation: the zero-shot CoT experiment uses a text-only exemplar without the corresponding image, which the authors explicitly classify as zero-shot despite the prompt format.
• Overlooking that MultiMedBench evaluation uses the exact same few-shot setup as training, meaning in-distribution performance is measured under identical prompt conditions.

Evidence (verbatim from paper)

Task-specific metrics were computed on the test split of each task and compared to prior SOTA specialist AI systems. We evaluated the accuracy across model scales by formulating this problem as a two-choice question answering task where the model was prompted (with a text-only one-shot exemplar) to generate a yes/no answer about the presence of TB in the input image.

Citation

@misc{tu2023towardsgeneralistbiomedicalai,
  title={Towards Generalist Biomedical AI},
  author={Tu et al. (2023)},
  year={2023},
  note={arXiv:2307.14334}
}

• arXiv: 2307.14334