qhjqhj00 video-class-agnostic-eval
Video Class Agnostic Segmentation Benchmark for Autonomous Driving — Siam et al. (2021) (arXiv:2103.11015, 2021)
What this evaluates
Evaluates a model's ability to segment moving and unknown objects in autonomous driving videos without relying on a closed set of known classes. It probes open-set and motion-based instance segmentation capabilities under varying data distributions and synthetic scenarios.
Datasets
- Cityscapes-VPS — total ?; splits: train (-1), test (-1)
- KITTI-MOTS — total ?; splits: train (-1), test (-1)
- Carla — total ?; splits: train (-1), test (-1)
Metrics
CAQ(primary) — range: percent- Class Agnostic Quality, measuring the segmentation quality of moving or unknown objects independent of their semantic class.
CA-IoU(primary) — range: percent- Intersection over Union computed exclusively on pixels belonging to unknown or class-agnostic objects during testing.
mIoU— range: percent- Mean Intersection over Union across all predicted semantic classes.
PQ— range: percent- Panoptic Quality, combining segmentation quality (SQ) and recognition quality (RQ) for all, thresholded, and static instances.
Input / output format
Input: Monocular video frames (typically resized to 1024x512) optionally accompanied by depth maps (simulated ground truth or estimated).
Output: Per-pixel segmentation masks for known semantic classes, moving instances, and a unified class-agnostic mask for unknown objects.
Scoring recipe
def compute_ca_iou(pred_mask, gt_mask, unknown_class_ids):
pred_unk = np.isin(pred_mask, unknown_class_ids)
gt_unk = np.isin(gt_mask, unknown_class_ids)
intersection = np.logical_and(pred_unk, gt_unk).sum()
union = np.logical_or(pred_unk, gt_unk).sum()
return (intersection / union) * 100 if union > 0 else 0.0
Common pitfalls
- Withholding specific classes (Person, Rider, Motorcycle, Bicycle) as 'unknown' during training but only evaluating CA-IoU on a subset (e.g., Motorcycle and Bicycle) during testing.
- Using different sets of unknown objects for training and testing to prevent overfitting, which complicates direct comparison across papers.
- Depth modality is not standardized: synthetic data uses ground-truth depth while real data uses estimated depth, affecting reproducibility.
Evidence (verbatim from paper)
The CA-IoU is reported only on the unknown objects used during testing, which are labelled in Figure 4.
Citation
@misc{siam2021video,
title={Video Class Agnostic Segmentation Benchmark for Autonomous Driving},
author={Siam et al. (2021)},
year={2021},
note={arXiv:2103.11015}
}
- arXiv: 2103.11015