Weather Robustness Od Eval

weather-robustness-od-eval

In Rain or Shine: Understanding and Overcoming Dataset Bias for Improving Robustness Against Weather Corruptions for Autonomous Vehicles — Marathe et al. (2022) (arXiv:2204.01062, 2022)

What this evaluates

Evaluates object detection robustness to adverse weather by measuring performance degradation when models trained on clear-weather datasets are tested on weather-corrupted images. It specifically quantifies dataset bias by comparing baseline in-distribution performance against out-of-distribution performance on the DAWN dataset.

Datasets

• DAWN — total 1000; splits: test (1000)
• Pascal VOC 2012 — total 21503; splits: train (16551), test (4952)
• Microsoft COCO 2017 — total 123000; splits: train (118000), val (5000)

Metrics

• mAP (primary) — range: percent
  • Mean Average Precision across the 4 target classes (car, bus, person, bicycle). Calculated as the arithmetic mean of the per-class AP scores.

Input / output format

Input: RGB images containing vehicles and pedestrians under varying weather conditions, paired with ground-truth bounding box coordinates and class labels.

Output: Predicted bounding box coordinates, confidence scores, and class labels for each detected object.

Scoring recipe

def compute_mAP(predictions, ground_truth):
    target_classes = ['car', 'bus', 'person', 'bicycle']
    aps = []
    for cls in target_classes:
        cls_preds = [p for p in predictions if p['class'] == cls]
        cls_gt = [g for g in ground_truth if g['class'] == cls]
        ap = calculate_AP(cls_preds, cls_gt, iou_thresh=0.5)
        aps.append(ap)
    return sum(aps) / len(aps) * 100

Common pitfalls

• Models are evaluated on only 4 specific classes (car, bus, person, bicycle) rather than the full dataset taxonomy, which can skew mAP if class distribution differs.
• The evaluation explicitly measures out-of-distribution robustness by testing on DAWN (adverse weather) after training on clear-weather data; reporting only in-distribution performance misses the core bias metric.
• AP values in the paper are reported as percentages (e.g., 74.95), not decimals, so scaling must be consistent when reproducing results.

Evidence (verbatim from paper)

For measuring the potential bias, the model was trained on the same training data as Step 1 and tested the performance on the 1000 adverse weather condition images of the DAWN dataset (Kenk and Hassaballah 2020). The model fails to perform satisfactorily on the corrupted images and reaches 3.75mAP with nearly 7 AP on most classes as shown in Table 1.

Citation

@misc{marathe2022weatherbias,
  title={In Rain or Shine: Understanding and Overcoming Dataset Bias for Improving Robustness Against Weather Corruptions for Autonomous Vehicles},
  author={Marathe et al. (2022)},
  year={2022},
  note={arXiv:2204.01062}
}

• arXiv: 2204.01062