Raddiagseg Eval

raddiagseg-eval

RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology — Chengrun Li et al. (2025) (arXiv:2510.18188, 2025)

What this evaluates

Evaluates a vision-language model's ability to perform joint radiological diagnosis, abnormality detection, and multi-target segmentation on X-ray and CT images. It probes the model's capacity for open-ended visual question answering, precise pixel-level mask generation, and robustness to label-imbalanced medical data.

Datasets

• RadDiagSeg-D — total 28800; splits: train (22000), test (6800)
• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for VQA, detection, and diagnosis tasks.
• Dice (primary) — range: [0, 1]
  • Dice coefficient measuring overlap between predicted and ground-truth masks: 2 * |A ∩ B| / (|A| + |B|). Used for segmentation quality.
• Recall — range: [0, 1]
  • True positive rate: TP / (TP + FN). Used for VQA and detection tasks.
• OpenQ-Acc — range: [0, 1]
  • Exact match accuracy for open-ended question answers.

Input / output format

Input: Radiological image (X-ray or CT slice) paired with a text prompt/question specifying the task (e.g., diagnosis query, detection instruction, or referring segmentation prompt).

Output: Text response (diagnosis label, detection yes/no, or open-ended answer) followed by segmentation mask tokens/coordinates. For segmentation tasks, pixel-level binary masks are generated.

Scoring recipe

def score(predictions, gold):
    f1s, recalls, dices = [], [], []
    for pred, g in zip(predictions, gold):
        pred_tok, gold_tok = set(pred[0].split()), set(g[0].split())
        if not pred_tok: f1s.append(0.0); recalls.append(0.0)
        else:
            prec = len(pred_tok & gold_tok) / len(pred_tok)
            rec = len(pred_tok & gold_tok) / len(gold_tok)
            f1s.append(2 * prec * rec / (prec + rec + 1e-6))
            recalls.append(rec)
        mask_p, mask_g = pred[1], g[1]
        inter = np.sum(mask_p & mask_g)
        dices.append(2 * inter / (np.sum(mask_p) + np.sum(mask_g) + 1e-6))
    return {'F1': np.mean(f1s), 'Recall': np.mean(recalls), 'Dice': np.mean(dices)}

Common pitfalls

• Label imbalance in RadDiagSeg-D necessitates using F1 instead of accuracy for diagnosis/detection; accuracy would be misleading.
• For joint VQA-Seg tasks, baseline models often fail to generate both text and masks, requiring task amputation or resulting in empty evaluation fields.
• 3D modality slices (CT/MRI) from the same volume must not leak across train/test splits to prevent data leakage and inflated performance.

Evidence (verbatim from paper)

For evaluation, we adopt F1 and Recall as metrics for the VQA tasks. We additionally document the Recall and Accuracy for open-ended questions. Following common practices, the Dice score is used to benchmark the quality of segmentation. For RadDiagSeg-D, given the label imbalance, we use F1 as the metric for detection and diagnosis.

Citation

@misc{li2025raddiagsegm,
  title={RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology},
  author={Chengrun Li et al. (2025)},
  year={2025},
  note={arXiv:2510.18188}
}

• arXiv: 2510.18188