Urban Syn Uda Eval

urban-syn-uda-eval

All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes — Gómez et al. (2023) (arXiv:2312.12176, 2023)

What this evaluates

Evaluates the utility of a semi-procedurally generated synthetic driving dataset (UrbanSyn) for unsupervised domain adaptation (UDA) in semantic segmentation. It probes whether combining multiple synthetic sources reduces the domain gap and improves pixel-level classification accuracy on real-world urban driving benchmarks.

Datasets

• UrbanSyn — total ?; splits: train (-1)
• GTAV — total ?; splits: train (-1)
• Synscapes — total ?; splits: train (-1)
• Cityscapes — total ?; splits: test (-1)
• BDD100K — total ?; splits: test (-1)
• Mapillary Vistas — total ?; splits: test (-1)

Metrics

• self-labeling accuracy (primary) — range: [0, 1]
  • Pixel-wise classification accuracy computed on real-world test images using pseudo-labels generated by the UDA model during co-training. Standard semantic segmentation UDA protocols typically report this as mean Intersection over Union (mIoU) across all classes.

Input / output format

Input: RGB images of urban driving scenes. Training data consists of synthetic images with pixel-level ground truth (semantic masks, instance masks, depth, bounding boxes, occlusion). Test data consists of unlabeled real-world driving images.

Output: Per-pixel semantic class predictions for each input image.

Scoring recipe

def compute_accuracy(predictions, ground_truth):
    correct = np.sum(predictions == ground_truth)
    total = ground_truth.size
    return correct / total
# Note: In practice, this is typically aggregated per-class and averaged (mIoU) to handle class imbalance.

Common pitfalls

• Assuming that simply concatenating multiple synthetic datasets automatically closes the domain gap without proper co-training or self-labeling strategies.
• Overlooking per-class performance disparities, as synthetic datasets may underrepresent rare classes (e.g., Bus, Rider) compared to real-world benchmarks.
• Confusing synthetic pixel-level ground truth with real-world annotations, which can lead to unrealistic evaluation expectations.

Evidence (verbatim from paper)

These labels are crucial for conducting synth-to-real UDA to develop semantic segmentation models without human labeling. We conduct comprehensive experiments in this section to demonstrate the importance of these labels. ... achieves state-of-the-art performance in unsupervised domain adaptation for semantic segmentation on real-world benchmarks (Cityscapes, BDD100K, Mapillary Vistas), demonstrating that diversity across synthetic sources outperforms single-dataset baselines by reducing domain gap and improving self-labeling accuracy via HRDA and co-training frameworks.

Citation

@misc{gomez2023urbansyn,
  title={All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes},
  author={Gómez et al. (2023)},
  year={2023},
  note={arXiv:2312.12176}
}

• arXiv: 2312.12176