Idd Aw Eval

idd-aw-eval

IDD-AW: A Benchmark for Safe and Robust Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather — Shaik et al. (2023) (arXiv:2311.14459, 2023)

What this evaluates

Evaluates the robustness and safety of semantic segmentation models for autonomous driving in unstructured traffic and adverse weather. It specifically probes whether models can correctly identify critical road elements and traffic participants when visual quality degrades due to rain, fog, snow, or low light.

Datasets

• IDD-AW — total 5000; splits: test (-1)

Metrics

• mIoU — range: percent
  • Standard mean Intersection-over-Union computed across all semantic classes by averaging per-class IoU scores.
• Safe mIoU (SmIoU) (primary) — range: percent (can be negative)
  • Modifies standard mIoU by applying a severity-based penalty for misclassifications on a designated set of important classes ($C_{imp}$). SmIoU equals mIoU when $C_{imp}$ is empty, but decreases (and can become negative) as more critical classes are added to penalize dangerous errors.

Input / output format

Input: Paired RGB and Near-Infrared (NIR) images of driving scenes. Models are evaluated using RGB-only, NIR-only, or stacked RGB+NIR inputs.

Output: Per-pixel semantic segmentation masks over a 4-level hierarchical label set covering road infrastructure, traffic participants, and roadside objects.

Scoring recipe

def compute_miou(pred, gt, classes):
    ious = []
    for c in classes:
        inter = np.sum((pred == c) & (gt == c))
        union = np.sum((pred == c) | (gt == c))
        ious.append(inter / union if union > 0 else 0.0)
    return np.mean(ious)

def compute_smIoU(pred, gt, classes, C_imp):
    base_miou = compute_miou(pred, gt, classes)
    penalty = 0.0
    for c in C_imp:
        error_pixels = np.sum((gt == c) & (pred != c))
        penalty += error_pixels
    total_imp = np.sum(np.isin(gt, C_imp))
    normalized_penalty = penalty / total_imp if total_imp > 0 else 0.0
    return base_miou - normalized_penalty

Common pitfalls

• Relying solely on standard mIoU hides dangerous misclassifications; SmIoU reveals significant safety gaps by dropping >15% or going negative for critical classes like bicycles and curbs.
• Ignoring the NIR modality severely underestimates model capability; RGB+NIR stacking yields >3% mIoU gains over RGB or NIR alone.
• Assuming pre-training on structured datasets (Cityscapes/ACDC) generalizes to unstructured traffic; performance drops below 50% mIoU on IDD-AW without domain-specific pre-training.

Evidence (verbatim from paper)

The SmIoU distribution is shifted to the lower side, indicating that it is finding a significant amount of dangerous mispredictions that are not accounted for in mIoU.

Citation

@misc{shaik2023iddaw,
  title={IDD-AW: A Benchmark for Safe and Robust Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather},
  author={Shaik et al. (2023)},
  year={2023},
  note={arXiv:2311.14459}
}

• arXiv: 2311.14459