Viewdelta Scd Eval

viewdelta-scd-eval

ViewDelta: Scaling Scene Change Detection through Text-Conditioning — Varghese et al. (2024) (arXiv:2412.07612, 2024)

What this evaluates

This evaluation probes a model's ability to perform scene change detection conditioned on natural language prompts. It measures how well the model distinguishes relevant semantic changes from nuisance variations across diverse domains (street-view, satellite, indoor) and handles viewpoint misalignments.

Datasets

• CSeg — total 500000; splits: test (-1)
• PSCD — total ?; splits: test (-1)
• SYSU-CD — total ?; splits: test (-1)
• VL-CMU-CD — total ?; splits: test (-1)

Metrics

• IoU (primary) — range: percent
  • Intersection over Union: IoU = TP / (TP + FP + FN). Measures the overlap between predicted and ground truth change masks.
• F1 score — range: percent
  • Harmonic mean of Precision and Recall: F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Recall — range: percent
  • True Positive Rate: Recall = TP / (TP + FN). Measures the proportion of actual changes correctly detected.
• Precision — range: percent
  • Positive Predictive Value: Precision = TP / (TP + FP). Measures the proportion of predicted changes that are correct.

Input / output format

Input: Bi-temporal image pairs accompanied by a natural language text prompt specifying the types of changes to detect.

Output: A segmentation mask (binary or multi-class) indicating changed regions, generated per image pair.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    pred = pred_mask.flatten()
    gt = gt_mask.flatten()
    tp = np.sum((pred == 1) & (gt == 1))
    fp = np.sum((pred == 1) & (gt == 0))
    fn = np.sum((pred == 0) & (gt == 1))
    iou = tp / (tp + fp + fn + 1e-8)
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    return {'IoU': iou, 'F1': f1, 'Recall': recall, 'Precision': precision}

Common pitfalls

• Prompting strategies are dataset-specific and must be adapted: class names for PSCD, comprehensive descriptions for SYSU-CD, or 'all' for unaligned variants.
• VL-CMU-CD contains label noise; using prompts that match consistent human labels is required for fair evaluation.
• Unaligned variants (Diff-1/Diff-2) test robustness to viewpoint shifts rather than standard change detection, requiring models to avoid spatial alignment assumptions.

Evidence (verbatim from paper)

We report standard change detection metrics including Intersection over Union (IoU), F1 score, Recall, and Precision. For each dataset, we adopt task-appropriate prompting strategies: dataset-specific class names for PSCD, a comprehensive change description for SYSU-CD, and a description of object categories for VL-CMU-CD.

Citation

@misc{varghese2024viewdelta,
  title={ViewDelta: Scaling Scene Change Detection through Text-Conditioning},
  author={Varghese et al. (2024)},
  year={2024},
  note={arXiv:2412.07612}
}

• arXiv: 2412.07612