Medical Vlm Eval

medical-vlm-eval

Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generation — Al-Hamadani (2025) (arXiv:2509.13590, 2025)

What this evaluates

Evaluates a vision-language model's ability to localize tumors in medical images and generate structured clinical reports. Probes spatial accuracy of coordinate prediction and the clinical utility/quality of automated radiology reports.

Datasets

• Unspecified medical imaging dataset — total ?; splits: test (-1)

Metrics

• positional_deviation (primary) — range: pixels
  • Absolute difference between predicted and reference tumor coordinates in pixels. Lower values indicate better spatial localization.
• confidence_score — range: 1 to 10
  • Certainty rating for each detected abnormality, scaled from 1 to 10 based on lesion clarity and boundary definition.

Input / output format

Input: Medical images (CT, MRI, X-ray, ultrasound) provided to a VLM.

Output: JSON object containing tumor location coordinates and metadata; structured clinical report with examination details, anatomical region, findings, and confidence scores (1-10).

Scoring recipe

def compute_coordinate_accuracy(pred_coords, ref_coords):
    return abs(pred_coords - ref_coords)  # in pixels

def compute_report_quality(report):
    # Multi-dimensional qualitative assessment: structural completeness, clinical accuracy, technical precision, adherence to standards
    return qualitative_rubric_score(report)

Common pitfalls

• Performance degrades in complex multi-pathology cases with overlapping abnormalities.
• Confidence scores may not perfectly correlate with actual diagnostic accuracy for subtle or ambiguous findings.
• Lack of standardized dataset and splits makes cross-study comparison difficult.

Evidence (verbatim from paper)

Overall positional deviations were reduced to within approximately ±80 pixels of the original location in the tested cases, demonstrating consistent and reliable performance across the evaluated dataset. Each identified abnormality receives a confidence score ranging from 1 to 10, offering clinicians quantitative measures of diagnostic certainty.

Citation

@misc{alhamadani2025intelligent,
  title={Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generation},
  author={Al-Hamadani (2025)},
  year={2025},
  note={arXiv:2509.13590}
}

• arXiv: 2509.13590