Refsgrs Eval

refsgrs-eval

RRSIS: Referring Remote Sensing Image Segmentation — Yuan et al. (2023) (arXiv:2306.08625, 2023)

What this evaluates

Evaluates a model's ability to segment specific objects in remote sensing images guided by natural language expressions, with a focus on accurately localizing small, scattered targets that are characteristic of aerial and satellite imagery.

Datasets

• RefSegRS — total 4420; splits: train (2172), val (431), test (1817)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean intersection-over-union. Obtained by averaging the IoU values (intersection area divided by union area) between predicted masks and ground truths across all test samples.
• oIoU — range: [0, 1]
  • Overall intersection-over-union. Calculated by taking the ratio of the total intersection area to the total union area across all test samples.
• Pr@0.5 — range: [0, 1]
  • Precision at threshold 0.5. Measures the proportion of test samples where the per-sample IoU is greater than or equal to 0.5.

Input / output format

Input: A remote sensing image paired with a natural language referring expression describing the target object(s) to segment.

Output: A pixel-wise binary mask (or probability map) matching the spatial dimensions of the input image, indicating the segmented region corresponding to the referring expression.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    ious = []
    total_intersection = 0
    total_union = 0
    for pred, gt in zip(predictions, ground_truths):
        inter = np.logical_and(pred, gt).sum()
        union = np.logical_or(pred, gt).sum()
        ious.append(inter / union if union > 0 else 0.0)
        total_intersection += inter
        total_union += union
    oIoU = total_intersection / total_union
    mIoU = np.mean(ious)
    Pr_at_05 = np.mean([iou >= 0.5 for iou in ious])
    return oIoU, mIoU, Pr_at_05

Common pitfalls

• oIoU heavily weights larger objects, potentially masking poor performance on small targets, while mIoU treats all objects equally regardless of size.
• Remote sensing images often contain very small foreground regions (<5% of pixels), making standard IoU thresholds highly sensitive to minor boundary errors or noise.
• Comparing methods implemented in different frameworks (e.g., TensorFlow vs. PyTorch) may introduce subtle implementation-level variance despite identical hyperparameters and training iterations.

Evidence (verbatim from paper)

The RefSegRS dataset consists of 4,420 image-language-label triples, with 2172 triples in the training set, 431 triples in the validation set, and 1817 triples in the test set. The commonly used metrics in referring image segmentation tasks include overall intersection-over-union (oIoU), mean intersection-over-union (mIoU), and precision at threshold values from 0.5 to 0.9. oIoU is calculated by taking the ratio of the total intersection area to the total union area across all test samples. mIoU is obtained by averaging IoU values between predicted masks and ground truths across all test samples. Precision at different threshold values measures the proportion of test samples that satisfy a specified IoU threshold.

Citation

@misc{yuan2023rrsis,
  title={RRSIS: Referring Remote Sensing Image Segmentation},
  author={Yuan et al. (2023)},
  year={2023},
  note={arXiv:2306.08625}
}

• arXiv: 2306.08625