Healthgpt Medical Vqa Generation Eval

healthgpt-medical-vqa-generation-eval

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation — Lin et al. (2025) (arXiv:2502.09838, 2025)

What this evaluates

Evaluates a medical vision-language model's ability to perform visual question answering (comprehension) and medical image synthesis (generation) on heterogeneous datasets. It probes the model's capacity to unify multiple downstream tasks using parameter-efficient fine-tuning without task interference.

Datasets

• VL-Health — total ?; splits: test (-1)
• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• PathVQA — total ?; splits: test (-1)
• IXI — total ?; splits: test (-1)
• SynthRAD2023 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions in visual comprehension tasks. Calculated as the number of exact or semantically matched predictions divided by the total number of test instances.
• SSIM — range: [-1, 1]
  • Structural Similarity Index Measure, evaluating luminance, contrast, and structure similarity between generated and ground-truth images. Values range from -1 to 1, with 1 indicating perfect similarity.
• PSNR — range: dB
  • Peak Signal-to-Noise Ratio, measuring the ratio between the maximum possible power of a signal and the power of corrupting noise. Higher values indicate better image quality.
• MSE — range: other
  • Mean Squared Error, calculating the average of the squares of the errors between predicted and actual pixel values. Lower values indicate better fidelity.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity, using deep network features to measure perceptual distance between images. Lower values indicate higher perceptual similarity.

Input / output format

Input: For comprehension: a medical image paired with a text question or instruction. For generation: a source medical image (for super-resolution or modality conversion) or a text instruction/report (for reconstruction).

Output: For comprehension: a text string containing the answer. For generation: a synthesized medical image matching the target resolution or modality.

Scoring recipe

def score_vqa(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if is_correct(p, g))
    return correct / len(golds) * 100

def score_generation(pred_imgs, gt_imgs):
    ssim = compute_ssim(pred_imgs, gt_imgs)
    psnr = compute_psnr(pred_imgs, gt_imgs)
    mse = compute_mse(pred_imgs, gt_imgs)
    lpips = compute_lpips(pred_imgs, gt_imgs)
    return {'SSIM': ssim, 'PSNR': psnr, 'MSE': mse, 'LPIPS': lpips}

Common pitfalls

• Confusing the three training strategies (LoRA, MoELoRA, H-LoRA) when comparing results, as they differ in rank settings, expert counts, and training time overhead.
• Assuming mixed training is directly comparable to the three-stage strategy without accounting for task interference and catastrophic forgetting explicitly noted in the ablation study.
• Overlooking that comprehension metrics are reported as percentages (accuracy) while generation metrics use different scales (SSIM/PSNR vs MSE/LPIPS), making direct cross-task comparison invalid.

Evidence (verbatim from paper)

Notably, most existing methods fail to fully leverage the prior knowledge of key structures in medical images, resulting in significant shortcomings in detail recovery. In contrast, our method significantly mitigates this issue. Specifically, HealthGPT-M3 excels in key metrics such as SSIM, PSNR, and ISE, achieving scores of 78.19, 32.76, and 34.47, respectively.

Citation

@misc{lin2025healthgpt,
  title={HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2502.09838}
}

• arXiv: 2502.09838