Cityscapes Eval

cityscapes-eval

The Cityscapes Dataset for Semantic Urban Scene Understanding — Cordts et al. (2016) (arXiv:1604.01685, 2016)

What this evaluates

Evaluates semantic scene understanding models on complex urban street scenes by measuring pixel-level classification accuracy and instance-level segmentation quality. It probes the model's ability to handle high-resolution imagery, diverse weather/lighting conditions, and fine-grained class distinctions in autonomous driving contexts.

Datasets

• Cityscapes — total ?; splits: train (-1), val (-1)

Metrics

• IoU (primary) — range: [0, 1]
  • Intersection over Union: the ratio of the area of overlap between the predicted mask and the ground truth mask to the area of their union. Computed per class and typically averaged to mIoU.
• iIoU — range: [0, 1]
  • Instance-normalized IoU, which adjusts the standard IoU calculation to account for instance-level variations and class imbalance in dense urban scenes.

Input / output format

Input: Single-frame monocular LDR images, often resized, cropped, or split into halves depending on the baseline's memory constraints.

Output: Per-pixel semantic class labels (dense masks) or instance-level segmentation masks/bounding boxes.

Scoring recipe

def compute_iou(pred_mask, gt_mask, class_id):
    pred = (pred_mask == class_id)
    gt = (gt_mask == class_id)
    intersection = np.logical_and(pred, gt).sum()
    union = np.logical_or(pred, gt).sum()
    if union == 0:
        return 1.0
    return intersection / union

def evaluate(dataset, model):
    ious = []
    for img, gt in dataset:
        pred = model(img)
        for cls in range(num_classes):
            ious.append(compute_iou(pred, gt, cls))
    return np.mean(ious)  # mIoU

Common pitfalls

• Ignoring void pixels during training causes gradients to be induced incorrectly, degrading performance.
• Input resolution and cropping strategies vary significantly across baselines, making direct comparison of inference time and accuracy difficult without normalization.
• Coarse annotations are used for pretraining or weak supervision, which can artificially inflate performance if not properly accounted for in the evaluation protocol.

Evidence (verbatim from paper)

Tables 9 and 11 list all individual class-level IoU scores for all control experiments and baselines. Tables 10 and 12 give the corresponding instance-normalized iIoU scores.

Citation

@misc{cordts2016cityscapes,
  title={The Cityscapes Dataset for Semantic Urban Scene Understanding},
  author={Cordts et al. (2016)},
  year={2016},
  note={arXiv:1604.01685}
}

• arXiv: 1604.01685