Vindrcxrvqa Eval

vindrcxrvqa-eval

VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learning — Nguyen et al. (2025) (arXiv:2511.00504, 2025)

What this evaluates

Evaluates a model's ability to answer clinical questions about chest X-rays and localize lesions via bounding boxes. It probes visual question answering, spatial grounding, and multi-task learning in a medical imaging context.

Datasets

• VinDr-CXR-VQA — total ?; splits: train (20880), val (659)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Prec * Rec) / (Prec + Rec). Computed on text-based lesion classification answers.
• mIoU (TP) — range: [0, 1]
  • Mean Intersection over Union computed only over true positive predictions (IoU ≥ 0.3).
• IoU≥0.5 (%) — range: percent
  • Percentage of validation images where at least one predicted bounding box achieves IoU ≥ 0.5 against ground truth.
• IoU≥0.3 (%) — range: percent
  • Percentage of validation images where at least one predicted bounding box achieves IoU ≥ 0.3 against ground truth.

Input / output format

Input: Chest X-ray image paired with a clinical question.

Output: Textual answer to the question, optionally containing bounding box coordinates embedded as special tokens ().

Scoring recipe

# VQA Classification
tp = sum(1 for p, g in zip(preds, golds) if p == g == "positive")
fp = sum(1 for p, g in zip(preds, golds) if p == "positive" and g != "positive")
fn = sum(1 for p, g in zip(preds, golds) if p != "positive" and g == "positive")
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

# Bounding Box Localization
ious = [compute_iou(pred_box, best_gt_box) for pred_box in pred_boxes]
tp_ious = [iou for iou in ious if iou >= 0.3]
mIoU_TP = mean(tp_ious) if tp_ious else 0
good_loc_pct = sum(1 for iou in ious if iou >= 0.5) / len(ious) * 100
acceptable_loc_pct = sum(1 for iou in ious if iou >= 0.3) / len(ious) * 100

Common pitfalls

• IoU threshold of 0.3 is used for detection metrics, but 0.5 is used for per-image 'good' localization; mixing these thresholds will skew results.
• mIoU is computed only over true positives (IoU ≥ 0.3), not over all predictions, which differs from standard object detection reporting.
• Bounding box coordinates are embedded as special tokens in the text output, requiring parsing before IoU calculation.

Evidence (verbatim from paper)

For VQA performance, we evaluate text-based lesion classification using Accuracy, Precision, Recall, and F1 score on the validation set (659 images, 5,471 ground-truth bounding boxes), comparing the pretrained baseline (zero-shot, no fine-tuning on our dataset) against our fine-tuned model. For bounding box localization, we adopt detection metrics with IoU threshold 0.3: a predicted box is a true positive (TP) if it overlaps any ground-truth box with IoU ≥ 0.3, otherwise a false positive (FP). Missed ground-truth boxes are false negatives (FN). We report Precision, Recall, F1, and mean IoU (mIoU) computed over TPs only.

Citation

@misc{nguyen2025vindrcxrvqa,
  title={VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learning},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2511.00504}
}

• arXiv: 2511.00504