Temmed Bench Eval

temmed-bench-eval

TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models — Zhang et al. (2025) (arXiv:2509.25143, 2025)

What this evaluates

Evaluates large vision-language models' ability to perform temporal reasoning on medical images by analyzing condition changes across multiple clinical visits. It probes capabilities in visual question answering, longitudinal clinical report generation, and selecting relevant image pairs based on temporal context.

Datasets

• TemMed-Bench — total 17000; splits: test (-1)

Metrics

• VQA Accuracy — range: percent
  • Percentage of instances where the model's predicted answer exactly matches the ground-truth answer.
• VQA F1 — range: percent
  • Token-level or exact-match F1 score between the predicted and ground-truth answers.
• Report Generation BLEU — range: percent
  • Standard BLEU n-gram overlap score between the generated report and the ground-truth clinical report.
• Report Generation ROUGE-L — range: percent
  • ROUGE-L recall/precision score based on the longest common subsequence between generated and ground-truth reports.
• Report Generation METEOR — range: percent
  • METEOR score incorporating synonymy, stemming, and paraphrase matching between generated and ground-truth reports.
• Image Selection Accuracy — range: percent
  • Percentage of instances where the model correctly selects the target image pair based on temporal condition changes.
• Avg. (primary) — range: percent
  • Arithmetic mean of VQA Accuracy, Report Generation BLEU, ROUGE-L, METEOR, and Image Selection Accuracy.

Input / output format

Input: A pair of medical images (historical and current visit) accompanied by a text prompt or question.

Output: Text answer (VQA), generated clinical report (Report Generation), or a selected image pair (Image Selection).

Scoring recipe

def score(predictions, golds):
    vqa_acc = sum(1 for p, g in zip(predictions['vqa'], golds['vqa']) if p == g) / len(golds['vqa'])
    vqa_f1 = compute_f1(golds['vqa'], predictions['vqa'])
    rep_bleu = compute_bleu(golds['report'], predictions['report'])
    rep_rouge = compute_rouge_l(golds['report'], predictions['report'])
    rep_meteor = compute_meteor(golds['report'], predictions['report'])
    img_acc = sum(1 for p, g in zip(predictions['img'], golds['img']) if p == g) / len(golds['img'])
    avg = (vqa_acc + rep_bleu + rep_rouge + rep_meteor + img_acc) / 5
    return {'vqa_acc': vqa_acc, 'vqa_f1': vqa_f1, 'rep_bleu': rep_bleu, 'rep_rouge': rep_rouge, 'rep_meteor': rep_meteor, 'img_acc': img_acc, 'avg': avg}

Common pitfalls

• Random-guess baselines are surprisingly high (33.3% for VQA and Image Selection), so models scoring near this threshold are effectively failing at temporal reasoning.
• Retrieval augmentation can degrade performance on the image-pair selection task due to attention splitting and retrieval noise when aligning with multiple target pairs.
• Medical fine-tuning can erode general reasoning capabilities, causing specialized medical LVLMs to underperform general-domain models on this benchmark.

Evidence (verbatim from paper)

For VQA, we use accuracy and F1 score as metrics. For report generation, following Jing et al. (2018) and Xia et al. (2025), we use BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), and METEOR (Banerjee & Lavie, 2005). For image-pair selection, accuracy is used.

Citation

@misc{zhang2025temmedbench,
  title={TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.25143}
}

• arXiv: 2509.25143