Medvilam Medical Bench Eval

medvilam-medical-bench-eval

MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation — Lijian Xu et al. (2024) (arXiv:2409.19684, 2024)

What this evaluates

Evaluates a multimodal LLM's ability to perform disease classification, visual grounding, and zero-shot reasoning across diverse medical imaging modalities (X-ray, CT, MRI, ultrasound, endoscopy, video) and tasks.

Datasets

• Chest X-ray (10 test sets from 5 public + 5 private) — total ?; splits: test (-1)
• ImageCAS, ASOCA, CCA200 — total ?; splits: test (-1)
• LPPolypVideo/SUN-SEG/CVC-12k — total ?; splits: test (-1)
• EndoVis18, LDPolyVideo — total ?; splits: test (-1)
• ISIC16, HAM10000 — total ?; splits: test (-1)
• TN3K, BUID — total ?; splits: test (-1)
• TBX11K, RSNA Pneumonia — total ?; splits: test (-1)
• Luna16, DeepLesion — total ?; splits: test (-1)
• ADNI, LGG — total ?; splits: test (-1)
• Object-CXR — total ?; splits: test (-1)

Metrics

• Accuracy (ACC) (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or bounding boxes out of total ground truth instances. Calculated as correct predictions divided by total samples.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between classes across all classification thresholds.
• Acc@0.5 — range: [0, 1]
  • Accuracy for referring bounding box detection where a predicted box is considered correct if its Intersection over Union (IoU) with the ground truth box is at least 0.5.

Input / output format

Input: Medical image (2D/3D/video) or audio paired with a natural language instruction or question (e.g., 'Is there a plaque on LM?', 'Where is the plaque on LM?', 'Is there anything foreign in this x-ray...?').

Output: Text response containing classification labels, yes/no answers, descriptive explanations, or bounding box coordinates formatted as 'center at [x, y, z], box length is [a, b, c]'.

Scoring recipe

def compute_acc(pred_labels, gt_labels):
    return sum(p == g for p, g in zip(pred_labels, gt_labels)) / len(gt_labels)

def compute_auc_f1(pred_scores, gt_labels):
    auc = roc_auc_score(gt_labels, pred_scores)
    f1 = f1_score(gt_labels, (pred_scores > 0.5).astype(int))
    return auc, f1

def compute_acc_at_iou(pred_boxes, gt_boxes, iou_thresh=0.5):
    correct = 0
    for p, g in zip(pred_boxes, gt_boxes):
        if calculate_iou(p, g) >= iou_thresh:
            correct += 1
    return correct / len(gt_boxes)

Common pitfalls

• Dataset splits and exact sizes are rarely disclosed; many are private or lack standard train/val/test splits, making reproducibility difficult.
• Acc@0.5 implies an IoU threshold of 0.5 for bounding box matching, but the exact matching logic (e.g., one-to-one vs. one-to-many assignment) is not specified.
• 3D grounding outputs use a specific coordinate format that requires precise parsing to evaluate correctly, and small lesion sizes in CT datasets cause near-zero accuracy regardless of model capability.

Evidence (verbatim from paper)

The overall accuracy (ACC) for plaque classification task was 30.1%/32.6%/34.5%, showing variations in performance across different coronary artery branches, likely due to differences in their morphological characteristics. For the visual grounding task of plaque localization, the overall ACC was 70.1%/73.2%/75.1%. Acc@0.5 is applied to evaluate methods.

Citation

@misc{xu2024medvilam,
  title={MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation},
  author={Lijian Xu et al. (2024)},
  year={2024},
  note={arXiv:2409.19684}
}

• arXiv: 2409.19684