Mtcityscapes 3d Eval

mtcityscapes-3d-eval

Joint 2D-3D Multi-Task Learning on Cityscapes-3D: 3D Detection, Segmentation, and Depth Estimation — Ye et al. (2023) (arXiv:2304.00971, 2023)

What this evaluates

Evaluates joint 2D-3D multi-task scene understanding on urban street imagery. It probes a model's ability to concurrently perform monocular 3D vehicle detection, 19-class semantic segmentation, and monocular depth estimation.

Datasets

• MTCityscapes-3D — total 3475; splits: train (2975), val (500)

Metrics

• mDS (primary) — range: [0, 1]
  • Mean detection score computed using the official Cityscapes-3D evaluation script.
• mIoU — range: [0, 1]
  • Mean Intersection over Union averaged across 19 semantic classes.
• RMSE — range: other
  • Root Mean Square Error between predicted and ground-truth depth values.

Input / output format

Input: RGB images at original resolution 1024×2048.

Output: Per image: 3D bounding boxes (location, rotation, dimensions, class, center-ness, direction) for vehicles; 19-class semantic segmentation map; monocular depth map.

Scoring recipe

def evaluate(preds, gold):
    # 3D Detection
    mDS = cityscapes3d_official_eval(preds['3d_boxes'], gold['3d_boxes'])
    # Segmentation
    ious = [intersection_over_union(preds['seg'][c], gold['seg'][c]) for c in range(19)]
    mIoU = sum(ious) / 19
    # Depth
    RMSE = sqrt(mean((preds['depth'] - gold['depth'])**2))
    return {'mDS': mDS, 'mIoU': mIoU, 'RMSE': RMSE}

Common pitfalls

• Evaluation is strictly performed at the original 1024×2048 resolution, despite training occurring at 768×1536.
• The 3D detection metric (mDS) relies on the official Cityscapes-3D script, which differs from standard KITTI/nuScenes protocols.
• Depth ground truth is derived from stereo cameras rather than LiDAR, introducing potential modality discrepancies.

Evidence (verbatim from paper)

The 3D vehicle detection task (3Ddet) utilizes the mean detection score (mDS) as its metric, using the official evaluation script provided by Cityscapes-3D. The models are assessed on the validation set for all tasks.

Citation

@misc{ye2023joint2d3d,
  title={Joint 2D-3D Multi-Task Learning on Cityscapes-3D: 3D Detection, Segmentation, and Depth Estimation},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2304.00971}
}

• arXiv: 2304.00971