Hulu Med Eval

hulu-med-eval

Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding — Songtao Jiang et al. (arXiv:2510.08668, 2025)

What this evaluates

Evaluates a unified vision-language model's capability to perform holistic medical understanding across text-only queries, 2D/3D medical images, and surgical videos. It probes visual question answering, radiology report generation, clinical reasoning, and multilingual medical dialogue.

Datasets

• MIMIC-CXR — total ?; splits: test (-1)
• CheXpert — total ?; splits: test (-1)
• IU X-ray — total ?; splits: test (-1)
• MedMNIST-2D — total ?; splits: test (-1)
• M3D — total ?; splits: test (-1)
• 3D-RAD — total ?; splits: test (-1)
• AMOS-MM — total ?; splits: test (-1)
• MedFrameQA — total ?; splits: test (-1)
• Cholec80-VQA — total ?; splits: test (-1)
• EndoVis18-VQA — total ?; splits: test (-1)
• PSI-AVA-VQA — total ?; splits: test (-1)
• SurgeryVideoQA — total ?; splits: test (-1)
• MMedBench — total ?; splits: test (-1)
• RareBench — total ?; splits: test (-1)
• HealthBench — total ?; splits: test (-1)
• MMLU-Pro-Med — total ?; splits: test (-1)
• MedXQA — total ?; splits: test (-1)
• Medbullets — total ?; splits: test (-1)
• SGPQA — total ?; splits: test (-1)
• MedMCQA — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• PubMedQA — total ?; splits: test (-1)
• MedXpertQA — total ?; splits: test (-1)
• MMMU-Med — total ?; splits: test (-1)
• OmniMedVQA — total ?; splits: test (-1)
• PMC-VQA — total ?; splits: test (-1)
• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• PathVQA — total ?; splits: test (-1)

Metrics

• RaTEScore (primary) — range: [0, 100]
  • A clinically oriented scoring metric that evaluates the clinical utility and accuracy of generated radiology reports, often validated against board-certified radiologist decisions.
• BLEU — range: [0, 1]
  • N-gram precision metric that measures the overlap between generated text and reference text, penalizing for brevity.
• ROUGE-L — range: [0, 1]
  • Recall-oriented metric based on the longest common subsequence between predicted and reference texts.
• METEOR — range: [0, 1]
  • Metric that aligns predicted and reference words using exact, stem, synonym, and paraphrase matches, then computes a weighted harmonic mean of precision and recall.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted answers or classifications out of the total number of instances.

Input / output format

Input: Textual instruction t concatenated with visual tokens from a 2D image, 3D volume, video sequence, or text-only input.

Output: Autoregressively generated textual response y.

Scoring recipe

def evaluate(predictions, gold):
    acc = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip()) / len(gold)
    bleu = compute_bleu(predictions, gold)
    rouge = compute_rouge(predictions, gold)
    meteor = compute_meteor(predictions, gold)
    rat_score = compute_rat_score(predictions, gold)
    return {'accuracy': acc, 'bleu': bleu, 'rouge': rouge, 'meteor': meteor, 'rat_score': rat_score}

Common pitfalls

• 3D volumetric data must be sliced into image sequences for baseline models, which can distort temporal/spatial context if not handled uniformly across all methods.
• Automated judges (e.g., ChatGPT-4o) were used for baselines lacking reported metrics, potentially introducing judge bias or inconsistency.
• Larger model sizes do not consistently yield better performance on medical report generation tasks; domain-specific pretraining is more critical than scale.

Evidence (verbatim from paper)

On MRG, we assessed Hulu-Med on three standard benchmarks—MIMIC-CXR, CheXpert, and IU X-ray—using both conventional natural language metrics (BLEU, ROUGE, METEOR) and the clinically oriented RaTEScore [[11], [73], [22], [27]] (Fig. [2]a-b).

Citation

@misc{jiang2025hulumed,
  title={Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding},
  author={Songtao Jiang et al.},
  year={2025},
  note={arXiv:2510.08668}
}

• arXiv: 2510.08668