Dronevehicle Eval

dronevehicle-eval

Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning — Sun et al. (2020) (arXiv:2003.02437, 2020)

What this evaluates

Evaluates aerial vehicle detection capability using aligned RGB and infrared image pairs. It specifically probes a model's ability to fuse cross-modal features and handle uncertainty in low-light or complex urban backgrounds.

Datasets

• DroneVehicle — total 28439; splits: train (17990), val (1469), test (8980); repo https://github.com/VisDrone/DroneVehicle

Metrics

• mAP (primary) — range: percent
  • Mean Average Precision across vehicle categories. A prediction is considered a true positive if the Intersection over Union (IoU) with the nearest ground-truth oriented bounding box exceeds 0.5. AP is computed per class and averaged.

Input / output format

Input: Aligned RGB and infrared aerial image pairs containing vehicles, with ground-truth oriented bounding box annotations for five vehicle categories (car, freight car, truck, bus, van).

Output: Oriented bounding boxes (OBB) with class labels and confidence scores for each detected vehicle instance.

Scoring recipe

def compute_mAP(predictions, ground_truths, iou_thresh=0.5):
    # predictions: list of (class, score, bbox)
    # ground_truths: list of (class, bbox)
    tp, fp = [], []
    for pred in sorted(predictions, key=lambda x: x[1], reverse=True):
        matched = False
        for gt in ground_truths:
            if pred[0] == gt[0] and not gt['used']:
                if compute_iou(pred[2], gt[2]) > iou_thresh:
                    tp.append(1); fp.append(0)
                    gt['used'] = True; matched = True; break
        if not matched: tp.append(0); fp.append(1)
    # Compute precision-recall curve per class, calculate AP, then average
    return average_precision(tp, fp)

Common pitfalls

• Using axis-aligned bounding boxes instead of oriented bounding boxes (OBB), which is required for aerial drone imagery.
• Applying the standard COCO IoU range (0.5:0.95) instead of the fixed 0.5 threshold explicitly stated in the protocol.
• Evaluating single-modality baselines and cross-modal fusion models on different splits or without identical experimental settings.

Evidence (verbatim from paper)

The standard metrics, Mean Average Precision (mAP) is adopted to evaluate the drone-based RGB-Infrared vehicle detection accuracy. The mAP measures the quality of bounding box predictions in the test set. Following [29], a prediction is considered as true positive if the IoU between the prediction and its nearest ground-truth annotation is larger than 0.5.

Citation

@misc{sun2020dronevehicle,
  title={Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning},
  author={Sun et al. (2020)},
  year={2020},
  note={arXiv:2003.02437}
}

• arXiv: 2003.02437