Medvision Eval

medvision-eval

MedVision: Dataset and Benchmark for Quantitative Medical Image Analysis — Yongcheng Yao et al. (2025) (arXiv:2511.18676, 2025)

What this evaluates

Evaluates vision-language models on quantitative medical image analysis, specifically anatomical structure detection, tumor/lesion size estimation, and angle/distance measurement. It probes the models' ability to perform precise spatial localization and numeric regression in a clinical context.

Datasets

• MedVision — total 30800000; splits: train (-1), test (-1)

Metrics

• IoU>0.5 (primary) — range: [0, 1]
  • Proportion of samples where the Intersection over Union between predicted and ground-truth bounding boxes exceeds 0.5. Reflects both instruction following and localization accuracy.
• Mean Relative Error (MRE) — range: [0, 1]
  • Average of |predicted - ground_truth| / ground_truth across all samples for tumor/lesion size and angle/distance tasks.
• Success Rate (SR) — range: [0, 1]
  • Proportion of samples where the model successfully generates valid numerical outputs (e.g., coordinates or measurements) as requested.

Input / output format

Input: A medical image (2D slice or 3D volume) paired with a text prompt containing anatomy/modality description, task instruction, format requirement, and physical spacing information (pixel size).

Output: Open-ended text response containing predicted bounding box coordinates, tumor/lesion dimensions, or angle/distance measurements, parsed into numerical values.

Scoring recipe

def score(predictions, golds, task_type):
    if task_type == 'detection':
        ious = [iou(pred_box, gold_box) for pred, gold in zip(predictions, golds)]
        return sum(1 for i in ious if i > 0.5) / len(ious)
    elif task_type in ['size', 'angle']:
        errors = [abs(p - g) / g for p, g in zip(predictions, golds)]
        return sum(errors) / len(errors)
    sr = sum(1 for p in predictions if is_valid_numerical(p)) / len(predictions)
    return sr

Common pitfalls

• Metrics are calculated only on successfully parsed outputs; models that fail to follow formatting instructions will have artificially low performance scores.
• Physical spacing (pixel size) must be correctly injected into prompts to match the model's preprocessing pipeline, otherwise real-world measurements will be incorrect.
• Detection tasks expect relative bounding box coordinates in [0,1], while size/angle tasks require absolute physical units; confusing these leads to systematic errors.

Evidence (verbatim from paper)

Model performance is evaluated with task-specific metrics. For detection tasks, we report the region-based recall, precision, F1 score, and intersection over union (IoU) for assessment of location accuracy. These metrics are calculated only on the successfully parsed outputs. To reflect the model’s ability to follow instructions, we also report the success rate (SR) of generating valid numerical outputs, such as coordinates. We also report IoU>0.5, the proportion of samples with IoU greater than 0.5 among all samples. Therefore, IoU>0.5 reflects both instruction following and localization ability. For T/L size and A/D measurement tasks, we report the mean relative error (MRE), mean absolute error (MAE), SR and MRE<k.

Citation

@misc{yao2025medvision,
  title={MedVision: Dataset and Benchmark for Quantitative Medical Image Analysis},
  author={Yongcheng Yao et al. (2025)},
  year={2025},
  note={arXiv:2511.18676}
}

• arXiv: 2511.18676