rsvg-eval
RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data — Zhan et al. (2022) (arXiv:2210.12634, 2022)
What this evaluates
This benchmark evaluates a model's ability to localize specific objects in remote sensing satellite imagery using natural language queries. It probes the model's robustness to scale variations, cluttered backgrounds, and multi-granularity textual descriptions common in aerial/satellite scenes.
Datasets
Metrics
Pr@0.5 (primary) — range: percent
- Percentage of image-query pairs where the predicted bounding box has an Intersection-over-Union (IoU) with the ground-truth box greater than or equal to 0.5.
Pr@0.6 — range: percent
- Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.6.
Pr@0.7 — range: percent
- Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.7.
Pr@0.8 — range: percent
- Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.8.
Pr@0.9 — range: percent
- Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.9.
meanIoU — range: [0, 1]
- Average IoU across all test samples: (1/M) * sum(I_t / U_t), where I_t and U_t are the intersection and union areas for sample t, and M is the dataset size.
cumIoU — range: [0, 1]
- Cumulative IoU across all test samples: sum(I_t) / sum(U_t), aggregating intersection and union areas before division.
Input / output format
Input: A remote sensing image and a natural language query describing a target object within the image.
Output: A single predicted bounding box (typically [x_min, y_min, x_max, y_max]) localizing the described object.
Scoring recipe
def score(predictions, gold_boxes):
ious = [compute_iou(p, g) for p, g in zip(predictions, gold_boxes)]
metrics = {}
for thresh in [0.5, 0.6, 0.7, 0.8, 0.9]:
metrics[f'Pr@{thresh}'] = sum(1 for i in ious if i >= thresh) / len(ious) * 100
metrics['meanIoU'] = sum(ious) / len(ious)
# cumIoU = sum(I_t) / sum(U_t). Since IoU = I_t/U_t, I_t = IoU * U_t.
metrics['cumIoU'] = sum(iou * union_area(p, g) for iou, p, g in zip(ious, predictions, gold_boxes)) / \
sum(union_area(p, g) for p, g in zip(predictions, gold_boxes))
return metrics
Common pitfalls
- IoU thresholds are strict in remote sensing due to small object sizes; Pr@0.9 often drops significantly compared to Pr@0.5.
- Cluttered backgrounds and scale variations in satellite imagery frequently cause false positives, making meanIoU and cumIoU more informative than binary precision at high thresholds.
- The dataset split is randomized by expression (40/10/50), not by image, which can lead to data leakage if the same image appears in train and test with different queries.
Evidence (verbatim from paper)
Given an RS image-query pair, the predicted bounding box is considered right if the intersection-over-union (IoU) with the ground-truth bounding box is above a threshold. In previous visual grounding works, a threshold of 0.5 is used as an accuracy metric. We report the metrics with IoU thresholds at 0.5, 0.6, 0.7, 0.8, and 0.9, termed as Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, and Pr@0.9, respectively. In addition, we follow the evaluation metrics of [[59]], including mean IoU and cumulative IoU (cumIoU), with the following equations: meanIoU = 1/M sum(I_t/U_t), and cumIoU = (sum I_t)/(sum U_t).
Citation
@misc{zhan2022rsvg,
title={RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data},
author={Zhan et al. (2022)},
year={2022},
note={arXiv:2210.12634}
}
1---2name: rsvg-eval3description: This benchmark evaluates a model's ability to localize specific objects in remote sensing satellite imagery using natural language queries. It probes the model's robustness to scale variations, cluttered backgrounds, and multi-granularity textual descriptions common in aerial/satellite scenes. Use when the user wants to benchmark on RSVGD, or asks about evaluating this task. Reports Pr@0.5.4metadata:5 source: https://skillmd.com/skills/qhjqhj00/rsvg-eval6 skill_kind: dataset_eval7 source_arxiv: 2210.126348 bibtex_key: zhan2022rsvg9 confidence: high10---1112# rsvg-eval1314> RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data — Zhan et al. (2022) (arXiv:2210.12634, 2022)1516## What this evaluates1718This benchmark evaluates a model's ability to localize specific objects in remote sensing satellite imagery using natural language queries. It probes the model's robustness to scale variations, cluttered backgrounds, and multi-granularity textual descriptions common in aerial/satellite scenes.1920## Datasets2122- **RSVGD** — total 38320; splits: train (-1), val (-1), test (-1); repo https://github.com/ZhanYang-nwpu/RSVG-pytorch2324## Metrics2526- `Pr@0.5` **(primary)** — range: percent27 - Percentage of image-query pairs where the predicted bounding box has an Intersection-over-Union (IoU) with the ground-truth box greater than or equal to 0.5.28- `Pr@0.6` — range: percent29 - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.6.30- `Pr@0.7` — range: percent31 - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.7.32- `Pr@0.8` — range: percent33 - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.8.34- `Pr@0.9` — range: percent35 - Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.9.36- `meanIoU` — range: [0, 1]37 - Average IoU across all test samples: (1/M) * sum(I_t / U_t), where I_t and U_t are the intersection and union areas for sample t, and M is the dataset size.38- `cumIoU` — range: [0, 1]39 - Cumulative IoU across all test samples: sum(I_t) / sum(U_t), aggregating intersection and union areas before division.4041## Input / output format4243**Input**: A remote sensing image and a natural language query describing a target object within the image.4445**Output**: A single predicted bounding box (typically [x_min, y_min, x_max, y_max]) localizing the described object.4647## Scoring recipe4849```python50def score(predictions, gold_boxes):51 ious = [compute_iou(p, g) for p, g in zip(predictions, gold_boxes)]52 metrics = {}53 for thresh in [0.5, 0.6, 0.7, 0.8, 0.9]:54 metrics[f'Pr@{thresh}'] = sum(1 for i in ious if i >= thresh) / len(ious) * 10055 metrics['meanIoU'] = sum(ious) / len(ious)56 # cumIoU = sum(I_t) / sum(U_t). Since IoU = I_t/U_t, I_t = IoU * U_t.57 metrics['cumIoU'] = sum(iou * union_area(p, g) for iou, p, g in zip(ious, predictions, gold_boxes)) / \58 sum(union_area(p, g) for p, g in zip(predictions, gold_boxes))59 return metrics60```6162## Common pitfalls6364- IoU thresholds are strict in remote sensing due to small object sizes; Pr@0.9 often drops significantly compared to Pr@0.5.65- Cluttered backgrounds and scale variations in satellite imagery frequently cause false positives, making meanIoU and cumIoU more informative than binary precision at high thresholds.66- The dataset split is randomized by expression (40/10/50), not by image, which can lead to data leakage if the same image appears in train and test with different queries.6768## Evidence (verbatim from paper)6970> Given an RS image-query pair, the predicted bounding box is considered right if the intersection-over-union (IoU) with the ground-truth bounding box is above a threshold. In previous visual grounding works, a threshold of 0.5 is used as an accuracy metric. We report the metrics with IoU thresholds at 0.5, 0.6, 0.7, 0.8, and 0.9, termed as Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, and Pr@0.9, respectively. In addition, we follow the evaluation metrics of [[59]], including mean IoU and cumulative IoU (cumIoU), with the following equations: meanIoU = 1/M sum(I_t/U_t), and cumIoU = (sum I_t)/(sum U_t).7172## Citation7374```bibtex75@misc{zhan2022rsvg,76 title={RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data},77 author={Zhan et al. (2022)},78 year={2022},79 note={arXiv:2210.12634}80}81```8283- arXiv: 2210.12634