# Valerie22 Eval

> This protocol evaluates the perceptual fidelity and cross-domain generalization capability of the VALERIE22 synthetic urban dataset by training a semantic segmentation model on it and testing on real-world automotive datasets. It specifically probes how dataset diversity (unique 3D assets) and training scale affect downstream perception performance. Use when the user wants to benchmark on VALERIE22, Cityscapes, A2D2, BDD100K, India Driving Dataset, Mapillary Vistas, or asks about evaluating this task. Reports mIoU.

- Skill: `qhjqhj00/valerie22-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/valerie22-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/valerie22-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/valerie22-eval

---


# valerie22-eval

> VALERIE22 -- A photorealistic, richly metadata annotated dataset of urban environments — Grau et al. (2023) (arXiv:2308.09632, 2023)

## What this evaluates

This protocol evaluates the perceptual fidelity and cross-domain generalization capability of the VALERIE22 synthetic urban dataset by training a semantic segmentation model on it and testing on real-world automotive datasets. It specifically probes how dataset diversity (unique 3D assets) and training scale affect downstream perception performance.

## Datasets

- **VALERIE22** — total ?; splits: train (-1)
- **Cityscapes** — total ?; splits: test (-1)
- **A2D2** — total ?; splits: test (-1)
- **BDD100K** — total ?; splits: test (-1)
- **India Driving Dataset** — total ?; splits: test (-1)
- **Mapillary Vistas** — total ?; splits: test (-1)

## Metrics

- `mIoU` **(primary)** — range: [0, 1]
  - Mean Intersection over Union computed over 11 shared semantic classes. Calculated as the average of (intersection of predicted and ground truth labels) / (union of predicted and ground truth labels) across all classes.

## Input / output format

**Input**: RGB images of urban street scenes.

**Output**: Pixel-wise semantic segmentation mask assigning one of 11 predefined classes (road, sidewalk, building, sky, car, truck, pole, traffic light, traffic sign, vegetation, person) to each pixel.

## Scoring recipe

```python
def compute_miou(pred_masks, gt_masks, num_classes=11):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        iou = intersection / union if union > 0 else 0.0
        ious.append(iou)
    return np.mean(ious)
```

## Common pitfalls

- Cross-domain evaluation requires strict 11-class mapping across synthetic and real datasets to ensure comparability.
- High frame count does not guarantee better generalization if asset diversity (unique 3D meshes) is low, leading to overfitting.
- Domain shift is significant when evaluating European-trained synthetic models on North American datasets like Mapillary Vistas.

## Evidence (verbatim from paper)

> To measure the performance of the task of semantic segmentation the mean Intersection over Union (mIoU) from the COCO semantic segmentation benchmark task is used [23]. The mIoU is denoted as the intersections between predicted semantic label classes and their corresponding ground truth divided by the union of the same, averaged over all classes.

## Citation

```bibtex
@misc{grau2023valerie22,
  title={VALERIE22 -- A photorealistic, richly metadata annotated dataset of urban environments},
  author={Grau et al. (2023)},
  year={2023},
  note={arXiv:2308.09632}
}
```

- arXiv: 2308.09632