Rwds He Eval

rwds-he-eval

Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery — Al-Emadi et al. (2025) (arXiv:2503.19202, 2025)

What this evaluates

Evaluates object detectors' robustness to real-world spatial domain shifts across hurricane-affected regions in satellite imagery. It measures how well models generalize from in-domain training data to out-of-distribution target domains without fine-tuning.

Datasets

• RWDS-HE — total ?; splits: train (-1), test (-1); repo https://github.com/RWGAI/RWDS

Metrics

• mAP (primary) — range: [0, 1]
  • MS-COCO AP metric calculated as the average precision over IoU thresholds ranging from 0.50 to 0.95 with a step size of 0.05.
• PD — range: percent
  • Performance Drop quantifies percentage degradation: 100 × (mAP_ID − mAP_OOD) / mAP_ID.
• H — range: [0, 1]
  • Harmonic Mean of ID and OOD mAP: (2 × mAP_OOD × mAP_ID) / (mAP_OOD + mAP_ID).

Input / output format

Input: 512×512 satellite image tiles cropped from raw imagery with a 0.2 overlap ratio.

Output: Bounding box coordinates and class labels for each detected object per tile.

Scoring recipe

def compute_metrics(preds, gold, id_mask, ood_mask):
    mAP_id = coco_eval(gold[id_mask], preds[id_mask], iou_range=(0.50, 0.95, 0.05))
    mAP_ood = coco_eval(gold[ood_mask], preds[ood_mask], iou_range=(0.50, 0.95, 0.05))
    PD = 100 * (mAP_id - mAP_ood) / mAP_id
    H = (2 * mAP_ood * mAP_id) / (mAP_ood + mAP_id)
    return {'mAP': mAP_id, 'mAP_OOD': mAP_ood, 'PD': PD, 'H': H}

Common pitfalls

• Using tuned hyperparameters instead of the default ones specified for each model, which contradicts the paper's instruction to mimic real-world conditions where tuning is impractical.
• Calculating PD or H using raw mAP values without ensuring the MS-COCO averaging convention (IoU 0.50-0.95 step 0.05) is applied consistently across ID and OOD splits.
• Treating the leave-one-domain-out setup as a standard train/val/test split without properly isolating the held-out domain as the sole OOD target.

Evidence (verbatim from paper)

We assess the performance of the object detectors using the standard mean Average Precision (mAP) metric which is commonly used in object detection applications. More specifically, we use the MS-COCO AP metric, which is calculated as the average over multiple IoU thresholds ranging from 0.50 to 0.95 with a stepsize of 0.05. Performance Drop (PD). A metric frequently used in the DG community for assessing the generalisability of classification tasks is the Performance Drop, which quantifies the percentage of performance degradation observed in the model when subjected to OOD data from a target domain. ... Harmonic Mean (H). To compare the ID and OOD performance of object detectors based on their mAP, we adopt the widely recognised Harmonic Mean as another evaluation metric.

Citation

@misc{al-emadi2025rwds,
  title={Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery},
  author={Al-Emadi et al. (2025)},
  year={2025},
  note={arXiv:2503.19202}
}

• arXiv: 2503.19202