Cross Domain Object Detection Eval

cross-domain-object-detection-eval

Unbiased Mean Teacher for Cross-domain Object Detection — Deng et al. (2020) (arXiv:2003.00707, 2020)

What this evaluates

Evaluates an object detection model's ability to generalize across domain shifts (e.g., real-to-artistic, clear-to-foggy, synthetic-to-real) using only labeled source data and unlabeled target data during training. It measures how well the model mitigates domain bias and adapts to unseen target distributions without target annotations.

Datasets

• PASCAL VOC 2007+2012 — total ?; splits: train (-1)
• Clipart1k — total 1000; splits: train (500), test (500)
• Watercolor2k — total 2000; splits: train (1000), test (1000)
• Cityscapes — total 3475; splits: train (2975), val (500)
• Foggy Cityscapes — total 3475; splits: train (2975), val (500)
• SIM10K — total 10000; splits: train (10000)

Metrics

• AP — range: percent
  • Average Precision computed per class over predicted bounding boxes and ground truth annotations, typically averaged across IoU thresholds or at a fixed threshold (e.g., 0.5).
• mAP (primary) — range: percent
  • Mean Average Precision calculated as the arithmetic mean of AP across all evaluated classes for a given dataset.

Input / output format

Input: Labeled images from the source domain and unlabeled images from the target domain. Images are resized so the shorter side is 600 pixels while preserving aspect ratio.

Output: Bounding box coordinates and class labels for each detected object per image.

Scoring recipe

def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    predictions.sort(key=lambda x: x.confidence, reverse=True)
    tp, fp = [], []
    for pred in predictions:
        if pred.class_id not in ground_truth: continue
        gt_box = ground_truth[pred.class_id].pop(0)
        if calculate_iou(pred.box, gt_box) >= iou_thresh:
            tp.append(1)
        else:
            fp.append(1)
    tp = np.cumsum(tp)
    fp = np.cumsum(fp)
    prec = tp / (tp + fp)
    rec = tp / len(ground_truth[pred.class_id])
    return np.trapz(prec, rec)

ap_per_class = {cls: compute_ap(preds[cls], gts[cls]) for cls in classes}
mAP = np.mean(list(ap_per_class.values())) * 100

Common pitfalls

• Using target domain ground truth annotations during training violates the strict unsupervised domain adaptation protocol.
• For Cityscapes and Foggy Cityscapes, the validation set serves as the test set; there is no separate held-out test split.
• Reporting only mAP without per-class AP obscures performance on rare or domain-sensitive classes (e.g., 'train' or 'truck' in foggy scenes).

Evidence (verbatim from paper)

Full annotations including the bounding boxes and the corresponding category labels of objects are available for the source domain training data, while the target domain only contains unlabeled images. Moreover, we can access only the unlabeled train set in the target domain, while the target domain test set is strictly held out during the training phase. We report the average precision (AP) of each class as well as the mean AP over all classes in Table 1 and Table 2 for object detection on the Clipart1k and Watercolor2k datasets, respectively.

Citation

@misc{deng2020unbiased,
  title={Unbiased Mean Teacher for Cross-domain Object Detection},
  author={Deng et al. (2020)},
  year={2020},
  note={arXiv:2003.00707}
}

• arXiv: 2003.00707