qhjqhj00 lae-1m-eval
Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community — Pan et al. (2024) (arXiv:2408.09110, 2024)
What this evaluates
Evaluates open-vocabulary and closed-set object detection capabilities on remote sensing imagery. It probes a model's ability to detect novel Earth-based objects without prior training on them, as well as its efficiency when fine-tuned with limited labeled data.
Datasets
- LAE-1M — total 1000000; splits: train (1000000); repo https://github.com/jaychempan/LAE-DINO
- DIOR — total ?; splits: test (-1)
- DOTAv2.0 — total ?; splits: test (-1)
- LAE-80C — total ?; splits: test (-1)
Metrics
mAP(primary) — range: [0, 100] percent- Mean Average Precision computed across all evaluated classes. Standard COCO-style calculation averaging AP over multiple IoU thresholds or a single threshold depending on the benchmark configuration.
AP50— range: [0, 100] percent- Average Precision calculated at an Intersection over Union (IoU) threshold of 0.5. Measures detection accuracy with relaxed localization requirements.
AP75— range: [0, 100] percent- Average Precision calculated at an IoU threshold of 0.75. Measures detection accuracy with strict localization requirements.
Input / output format
Input: Remote sensing satellite or aerial images. For open-vocabulary evaluation, text prompts or class names are provided as queries. For closed-set fine-tuning, images are paired with ground-truth bounding boxes and class labels.
Output: A list of predicted bounding boxes (coordinates) and associated class labels with confidence scores for each input image.
Scoring recipe
def compute_ap(predictions, ground_truth, iou_thresh=0.5):
# Sort predictions by confidence descending
# Match each prediction to the highest IoU ground truth box
# Classify as TP if IoU >= iou_thresh and box not yet matched, else FP
# Compute precision-recall curve and interpolate AP
return ap
def compute_mAP(predictions_per_class, ground_truth_per_class, iou_thresh=0.5):
aps = [compute_ap(preds, gts, iou_thresh) for preds, gts in predictions_per_class.items()]
return sum(aps) / len(aps) * 100
Common pitfalls
- DOTAv2.0 results must strictly use horizontal bounding boxes, not rotated ones, as explicitly required by the authors.
- Open-vocabulary pre-training on natural scene datasets (e.g., O365, GoldG) performs poorly on remote sensing benchmarks due to a substantial domain gap; direct comparison without domain adaptation is misleading.
- LAE-1M construction uses class-dependent sampling rates (0.4 for most classes, 0.2 for Xview) to balance distribution, which affects training data composition and should be noted when reproducing.
Evidence (verbatim from paper)
The $mAP$, $AP_{50}$, and $AP_{75}$ are used as the evaluation metrics. Note that the results of DOTAv2.0 are all based on horizontal detection boxes for building a foundational location detector.
Citation
@misc{pan2024locate,
title={Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community},
author={Pan et al. (2024)},
year={2024},
note={arXiv:2408.09110}
}
- arXiv: 2408.09110