Tunevlseg Eval

tunevlseg-eval

TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models — Adhikari et al. (2024) (arXiv:2410.05239, 2024)

What this evaluates

Evaluates the robustness and zero-shot adaptation of vision-language segmentation models under prompt tuning across diverse medical and natural domain datasets. It probes how different prompt tuning strategies and prompt depths handle domain shifts and varying class counts.

Datasets

• Kvasir-SEG — total ?; splits: train (800), val (100), test (100)
• ClinicDB — total ?; splits: train (490), val (61), test (61)
• BKAI — total ?; splits: train (800), val (100), test (100)
• ISIC 2016 — total ?; splits: train (810), val (90), test (379)
• DFU 2022 — total ?; splits: train (1600), val (200), test (200)
• CAMUS — total ?; splits: train (4800), val (600), test (600)
• BUSI — total ?; splits: train (624), val (78), test (78)
• CheXlocalize — total ?; splits: train (1279), val (446), test (452)
• Cityscapes — total ?; splits: train (34723), val (6005), test (-1)
• PascalVOC — total ?; splits: train (2170), val (2148), test (-1)

Metrics

• Dice score (primary) — range: [0, 1]
  • Computed as 2 * |A ∩ B| / (|A| + |B|) between predicted and ground truth masks. The paper uses Dice Loss for training, implying Dice score for evaluation.

Input / output format

Input: Resized images (416x416 for CRIS, 352x352 for CLIPSeg) with foreground class name(s) provided as text prompts. Images are normalized and lightly augmented during training.

Output: Binary or multi-class segmentation mask per image, corresponding to the foreground class(es) specified in the prompt.

Scoring recipe

def compute_dice(pred, gt):
    intersection = np.sum(pred * gt)
    union = np.sum(pred) + np.sum(gt)
    return (2.0 * intersection) / (union + 1e-6)
# Average Dice score across test set

Common pitfalls

• Significant domain shift between medical and open-domain datasets requires careful prompt depth tuning.
• Hyperparameters are swept per dataset (20 runs each), leading to high variance across datasets.
• Medical datasets have few classes (1-10) while open-domain have ~20, affecting context learner performance.
• Only foreground class names are used for prompts, ignoring background or multi-class labels.

Evidence (verbatim from paper)

For our empirical analysis of medical datasets, we utilize eight of the datasets and their splits provided by Poudel et al. ... For the loss function, we used a combined loss of Dice Loss and Binary Cross Entropy loss.

Citation

@misc{adhikari2024tunevlseg,
  title={TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models},
  author={Adhikari et al. (2024)},
  year={2024},
  note={arXiv:2410.05239}
}

• arXiv: 2410.05239