Medvidbench Eval

medvidbench-eval

MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding — Su et al. (2025) (arXiv:2512.06581, 2025)

What this evaluates

Evaluates heterogeneous medical video understanding across classification, grounding, and captioning tasks. Probes a model's ability to perform clinical safety checks, predict surgical actions, assess skills, localize temporal/spatial events, and generate precise medical video descriptions.

Datasets

• MedVidBench (Standard) — total 51505; splits: train (45260), test (6245)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions for classification tasks (CVS, NA, SA). Computed as correct predictions divided by total samples.
• mIoU — range: [0, 1]
  • Mean Intersection over Union for temporal (TAG) and spatiotemporal (STG) grounding tasks. Computed as the intersection of predicted and ground-truth bounding boxes divided by their union, averaged across samples at IoU thresholds of 0.3 and 0.5.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for Dense Video Captioning (DVC), evaluating the overlap between generated and ground-truth captions.
• LLM judge score — range: other
  • Comparative similarity score generated by a medical LLM judge evaluating five clinical dimensions: terminology precision, instrument/anatomy identification, specificity, procedural context, and action accuracy.

Input / output format

Input: Video frames (sampled at 1 FPS) paired with a text prompt/question specifying the task (e.g., region captioning, action prediction, safety assessment).

Output: Text response: clinical description, action label, safety status, or structured caption depending on the task.

Scoring recipe

def compute_metric(task, pred, gold):
    if task in ['CVS', 'NA', 'SA']:
        return 'accuracy', 1.0 if pred == gold else 0.0
    elif task in ['STG', 'TAG']:
        iou = intersection(pred_box, gold_box) / union(pred_box, gold_box)
        return 'mIoU', iou
    elif task == 'DVC':
        return 'F1 score', f1_score(pred_caption, gold_caption)
    elif task in ['DVC', 'VS', 'RC']:
        return 'LLM judge score', medical_llm_judge.evaluate(pred_caption, gold_caption)
    return None, None

Common pitfalls

• Removing cross-dataset reward normalization causes catastrophic training collapse due to high magnitude differences between tasks.
• Using standard semantic similarity metrics for captioning inflates scores without capturing clinical terminology precision or anatomical accuracy.
• The LargeScale version has natural task imbalance favoring captioning, which can skew scaling law experiments if not accounted for.

Evidence (verbatim from paper)

Video-level tasks include Video Summarization (VS) measured by LLM judge score; Critical View of Safety (CVS) assessed by accuracy; Next Action Prediction (NA) evaluated by accuracy; and Skill Assessment (SA) measured by accuracy. Segment-level tasks include Temporal Action Grounding (TAG) measured by mean IoU at thresholds 0.3 and 0.5; Dense Video Captioning (DVC) evaluated by LLM judge score and F1 score; and Region Captioning (RC) assessed by LLM judge score. Frame-level tasks include Spatiotemporal Grounding (STG) measured by mIoU.

Citation

@misc{su2025medgrpo,
  title={MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2512.06581}
}

• arXiv: 2512.06581