Fleming Vl Medical Eval

fleming-vl-medical-eval

Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs — Shu et al. (2025) (arXiv:2511.00916, 2025)

What this evaluates

Evaluates a multimodal LLM's ability to perform visual reasoning across heterogeneous medical modalities (2D images, 3D volumes, videos) and generate clinical reports. It probes diagnostic accuracy, cross-modal generalization, temporal understanding, and structured medical knowledge integration.

Datasets

• OmniMedVQA — total ?; splits: test (-1)
• PMC-VQA — total 227000; splits: test (-1)
• VQA-RAD — total 315; splits: test (-1)
• PathVQA — total 32799; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• MIMIC-CXR — total 377110; splits: test (-1)
• IU-Xray — total 7470; splits: test (-1)
• M3D-VQA — total 13791; splits: test (-1)
• MedVideoBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered multiple-choice or open-ended questions. Computed as the ratio of exact matches between predicted and ground-truth answers.
• ROUGE-L (primary) — range: percent
  • Longest common subsequence overlap between generated and reference text, scaled by 100 in reported tables.
• CIDEr (primary) — range: percent
  • Consensus-based image description evaluation using TF-IDF weighted n-grams, scaled by 100.
• RaTE — range: percent
  • Clinical fidelity metric based on Radiology Text Embedding similarity between generated and reference reports.
• Semb — range: percent
  • Semantic clinical fidelity metric measuring embedding-level alignment between predictions and gold reports.
• RadCliQ-1 — range: percent
  • Clinical quality scoring metric evaluating the diagnostic accuracy and clinical coherence of generated reports.

Input / output format

Input: Medical image (2D/3D/video frame) or video sequence paired with a natural language question or instruction to generate a report.

Output: Natural language text: either a short answer/option for VQA, or a structured clinical report (findings, impressions, diagnosis).

Scoring recipe

def evaluate(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower()) / len(golds)
    rouge_l = rouge_l_score(predictions, golds) * 100
    cider = cider_score(predictions, golds) * 100
    rate = clinical_embedding_similarity(predictions, golds, model='RaTE') * 100
    semb = clinical_embedding_similarity(predictions, golds, model='Semb') * 100
    radcliq = clinical_quality_score(predictions, golds, model='RadCliQ-1') * 100
    return {'accuracy': acc, 'ROUGE-L': rouge_l, 'CIDEr': cider, 'RaTE': rate, 'Semb': semb, 'RadCliQ-1': radcliq}

Common pitfalls

• Tables scale all scores by a factor of 100 for clarity; evaluation code must multiply or divide accordingly to match reported values.
• MedVideoBench is a newly proposed benchmark by the authors; external reproduction requires accessing their supplementary code or data release.
• Clinical fidelity metrics (RaTE, Semb, RadCliQ-1) rely on specific pretrained clinical NLP models that must be correctly initialized and aligned with the evaluation script.

Evidence (verbatim from paper)

For holistic video understanding, we employ ROUGE-L and CIDEr metrics to evaluate the quality and clinical fidelity of generated video summaries, while multiple-choice questions are evaluated using accuracy.

Citation

@misc{shu2025flemingvl,
  title={Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs},
  author={Shu et al. (2025)},
  year={2025},
  note={arXiv:2511.00916}
}

• arXiv: 2511.00916