Rsvg Eval

rsvg-eval

RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data — Zhan et al. (2022) (arXiv:2210.12634, 2022)

What this evaluates

This benchmark evaluates a model's ability to localize specific objects in remote sensing satellite imagery using natural language queries. It probes the model's robustness to scale variations, cluttered backgrounds, and multi-granularity textual descriptions common in aerial/satellite scenes.

Datasets

• RSVGD — total 38320; splits: train (-1), val (-1), test (-1); repo https://github.com/ZhanYang-nwpu/RSVG-pytorch

Metrics

• Pr@0.5 (primary) — range: percent
  • Percentage of image-query pairs where the predicted bounding box has an Intersection-over-Union (IoU) with the ground-truth box greater than or equal to 0.5.
• Pr@0.6 — range: percent
  • Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.6.
• Pr@0.7 — range: percent
  • Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.7.
• Pr@0.8 — range: percent
  • Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.8.
• Pr@0.9 — range: percent
  • Percentage of image-query pairs where the predicted bounding box has an IoU with the ground-truth box greater than or equal to 0.9.
• meanIoU — range: [0, 1]
  • Average IoU across all test samples: (1/M) * sum(I_t / U_t), where I_t and U_t are the intersection and union areas for sample t, and M is the dataset size.
• cumIoU — range: [0, 1]
  • Cumulative IoU across all test samples: sum(I_t) / sum(U_t), aggregating intersection and union areas before division.

Input / output format

Input: A remote sensing image and a natural language query describing a target object within the image.

Output: A single predicted bounding box (typically [x_min, y_min, x_max, y_max]) localizing the described object.

Scoring recipe

def score(predictions, gold_boxes):
    ious = [compute_iou(p, g) for p, g in zip(predictions, gold_boxes)]
    metrics = {}
    for thresh in [0.5, 0.6, 0.7, 0.8, 0.9]:
        metrics[f'Pr@{thresh}'] = sum(1 for i in ious if i >= thresh) / len(ious) * 100
    metrics['meanIoU'] = sum(ious) / len(ious)
    # cumIoU = sum(I_t) / sum(U_t). Since IoU = I_t/U_t, I_t = IoU * U_t.
    metrics['cumIoU'] = sum(iou * union_area(p, g) for iou, p, g in zip(ious, predictions, gold_boxes)) / \
                        sum(union_area(p, g) for p, g in zip(predictions, gold_boxes))
    return metrics

Common pitfalls

• IoU thresholds are strict in remote sensing due to small object sizes; Pr@0.9 often drops significantly compared to Pr@0.5.
• Cluttered backgrounds and scale variations in satellite imagery frequently cause false positives, making meanIoU and cumIoU more informative than binary precision at high thresholds.
• The dataset split is randomized by expression (40/10/50), not by image, which can lead to data leakage if the same image appears in train and test with different queries.

Evidence (verbatim from paper)

Given an RS image-query pair, the predicted bounding box is considered right if the intersection-over-union (IoU) with the ground-truth bounding box is above a threshold. In previous visual grounding works, a threshold of 0.5 is used as an accuracy metric. We report the metrics with IoU thresholds at 0.5, 0.6, 0.7, 0.8, and 0.9, termed as Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, and Pr@0.9, respectively. In addition, we follow the evaluation metrics of [[59]], including mean IoU and cumulative IoU (cumIoU), with the following equations: meanIoU = 1/M sum(I_t/U_t), and cumIoU = (sum I_t)/(sum U_t).

Citation

@misc{zhan2022rsvg,
  title={RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data},
  author={Zhan et al. (2022)},
  year={2022},
  note={arXiv:2210.12634}
}

• arXiv: 2210.12634