Bimedx2 Medical Eval

bimedx2-medical-eval

BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities — Mullappilly et al. (2024) (arXiv:2412.07769, 2024)

What this evaluates

Evaluates a bilingual (Arabic-English) large multimodal model's ability to understand diverse medical imaging modalities, answer visual questions, and generate or summarize clinical reports. It probes factual accuracy, clinical relevance, and linguistic quality across text-only, visual-question-answering, and report-generation tasks.

Datasets

• BiMed-MBench — total 386; splits: test (386)
• Rad-VQA — total ?; splits: (unstated)
• SLAKE — total ?; splits: (unstated)
• Path-VQA — total ?; splits: (unstated)
• MIMIC-CXR — total ?; splits: (unstated)
• MIMIC-III — total ?; splits: (unstated)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match ratio for closed-ended VQA questions. Computed as correct predictions divided by total questions.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for open-ended VQA questions. Measures overlap between predicted and gold token sets.
• BLEU-1 — range: [0, 1]
  • Unigram precision score measuring lexical overlap between generated and reference text.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence-based recall/precision score capturing sentence-level structural similarity.
• F1-RadGraph (primary) — range: [0, 1]
  • F1 score computed over extracted medical entities and relations using the RadGraph pipeline, measuring clinical entity preservation.
• RadCliQ — range: [0, 1]
  • Clinical quality metric scoring the diagnostic correctness and completeness of generated radiology reports.
• CheXbert similarity — range: [0, 1]
  • Vector similarity score between predicted and ground-truth clinical labels extracted via the CheXbert classifier.
• GPT-4o score (primary) — range: [1, 10]
  • LLM-as-a-judge rating on a 1-10 scale evaluating correctness, helpfulness, relevance, accuracy, and level of detail against a reference or upper-bound answer.

Input / output format

Input: Medical images (CXR, MRI, histology, CT, gross pathology) paired with text prompts/questions. For report tasks: one or more radiology images + prompt 'Please caption this scan with findings and impressions.' or text findings + prompt 'Summarize the findings'.

Output: Natural language responses: direct answers to VQA prompts, or structured medical report sections (findings/impressions) for generation/summarization tasks.

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type == 'vqa_closed':
        return exact_match(predictions, gold)
    elif task_type == 'vqa_open':
        return f1_score(predictions, gold)
    elif task_type in ['report_gen', 'report_summ']:
        lex = compute_bleu_rouge_meteor(predictions, gold)
        clin = compute_radgraph_f1(predictions, gold)
        clin += compute_chexbert_similarity(predictions, gold)
        clin += compute_radcliq(predictions, gold)
        return lex, clin
    elif task_type == 'bimem_bench':
        ref = gold if gold else generate_upper_bound(predictions)
        return gpt4o_evaluate(pred=predictions, ref=ref,
                              criteria=['correctness','helpfulness','relevance','accuracy','detail'])
    return None

Common pitfalls

• Relying solely on lexical metrics (BLEU/ROUGE) for clinical reports fails to capture diagnostic intent and clinical accuracy.
• Using cascaded translation pipelines for Arabic evaluation yields inferior results compared to dedicated bilingual instruction tuning.
• GPT-4o evaluation requires generating a high-quality upper-bound reference first; poor references bias the 1-10 scoring scale.

Evidence (verbatim from paper)

In both report generation and summarization tasks, relying solely on lexical metrics such as BLEU, ROUGE-L, and METEOR is insufficient, as these do not guarantee clinical accuracy. Therefore, we complement them with clinically-informed metrics such as F1-RadGraph, CheXbert similarity, and RadCliQ.

Citation

@misc{mullappilly2024bimedx2,
  title={BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities},
  author={Mullappilly et al. (2024)},
  year={2024},
  note={arXiv:2412.07769}
}

• arXiv: 2412.07769