Valerie22 Eval

valerie22-eval

VALERIE22 -- A photorealistic, richly metadata annotated dataset of urban environments — Grau et al. (2023) (arXiv:2308.09632, 2023)

What this evaluates

This protocol evaluates the perceptual fidelity and cross-domain generalization capability of the VALERIE22 synthetic urban dataset by training a semantic segmentation model on it and testing on real-world automotive datasets. It specifically probes how dataset diversity (unique 3D assets) and training scale affect downstream perception performance.

Datasets

• VALERIE22 — total ?; splits: train (-1)
• Cityscapes — total ?; splits: test (-1)
• A2D2 — total ?; splits: test (-1)
• BDD100K — total ?; splits: test (-1)
• India Driving Dataset — total ?; splits: test (-1)
• Mapillary Vistas — total ?; splits: test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union computed over 11 shared semantic classes. Calculated as the average of (intersection of predicted and ground truth labels) / (union of predicted and ground truth labels) across all classes.

Input / output format

Input: RGB images of urban street scenes.

Output: Pixel-wise semantic segmentation mask assigning one of 11 predefined classes (road, sidewalk, building, sky, car, truck, pole, traffic light, traffic sign, vegetation, person) to each pixel.

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes=11):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        iou = intersection / union if union > 0 else 0.0
        ious.append(iou)
    return np.mean(ious)

Common pitfalls

• Cross-domain evaluation requires strict 11-class mapping across synthetic and real datasets to ensure comparability.
• High frame count does not guarantee better generalization if asset diversity (unique 3D meshes) is low, leading to overfitting.
• Domain shift is significant when evaluating European-trained synthetic models on North American datasets like Mapillary Vistas.

Evidence (verbatim from paper)

To measure the performance of the task of semantic segmentation the mean Intersection over Union (mIoU) from the COCO semantic segmentation benchmark task is used [23]. The mIoU is denoted as the intersections between predicted semantic label classes and their corresponding ground truth divided by the union of the same, averaged over all classes.

Citation

@misc{grau2023valerie22,
  title={VALERIE22 -- A photorealistic, richly metadata annotated dataset of urban environments},
  author={Grau et al. (2023)},
  year={2023},
  note={arXiv:2308.09632}
}

• arXiv: 2308.09632