Grin Drive Eval

grin-drive-eval

GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions — Katsumata et al. (2025) (arXiv:2508.21102, 2025)

What this evaluates

Evaluates a model's ability to generate polygon segmentation masks for generalized referring navigable regions in autonomous driving scenes based on natural language navigation instructions. It specifically probes handling of single-target, multi-target, and no-target scenarios without biasing towards trivial existence predictions.

Datasets

• GRiN-Drive — total 17114; splits: train (14973), val (1413), test (758)

Metrics

• msIoU (primary) — range: [0, 1]
  • Mean of sIoU@k across N samples. sIoU@k = min(k * IoU(pred, gt), 1) if true positive, 1 if true negative, 0 if false positive/negative. Averaged over thresholds k=1..1/K.
• P@0.1 — range: percent
  • Percentage of samples where the predicted mask IoU exceeds 0.1 for true positive cases.
• Acc. — range: percent
  • Accuracy of the target existence prediction (TP/TN/FP/FN classification).

Input / output format

Input: RGB image and a natural language navigation instruction.

Output: One or more polygon masks representing the navigable region(s) corresponding to the instruction, along with an implicit existence prediction (target present/absent).

Scoring recipe

def compute_msIoU(pred_masks, gt_masks, K=10):
    scores = []
    for pred, gt in zip(pred_masks, gt_masks):
        iou = calculate_iou(pred, gt)
        is_tp = iou > 0.0 and gt is not empty
        is_tn = iou == 0.0 and gt is empty
        if is_tp:
            k_scores = [min((k/K) * iou, 1.0) for k in range(1, K+1)]
            scores.append(sum(k_scores) / K)
        elif is_tn:
            scores.append(1.0)
        else:
            scores.append(0.0)
    return sum(scores) / len(scores)

Common pitfalls

• Using standard gIoU or raw IoU metrics heavily biases evaluation toward trivial 'no-target' predictions, as always predicting no target yields a deceptively high score (~0.33 gIoU).
• Failing to account for multi-target or no-target cases equally, as standard metrics do not penalize missing multiple targets or falsely predicting targets in empty scenes with the same weight as single-target errors.

Evidence (verbatim from paper)

We propose the new metric msIoU to evaluate single-target, multi-target, and no-target samples without biases. By contrast, most existing metrics yield biased evaluations: a correct no-target prediction yields an IoU of 1.0, whereas even accurate target segmentations yield scores lower than 1.0. Therefore, the metric favors trivial solutions focusing on target-existence classification, which is weighted more than mask generation. Indeed, in our task, even a trivial solution that always predicts “no target” can achieve a gIoU score of 0.33, which is deceptively high, surpassing the human performance of msIoU 0.17 on target samples in the test set of the GRiN-Drive benchmark. To address this, msIoU assigns a score of 1 to samples with IoU exceeding a threshold K, while normalizing IoU scores based on K for those below the threshold, as shown in Equation [3]. By further averaging sIoU@k where (k=1, …, 1/K), msIoU achieves a balanced evaluation for both target and no-target samples. The details of the remaining evaluation metrics are explained in the supplementary materials. msIoU is defined as follows: msIoU = mean(1/N sum_{i=1}^N sIoU@k_i), sIoU@k_i = min(k * IoU(y_hat_i, y_i), 1) if

Citation

@misc{katsumata2025gennav,
  title={GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions},
  author={Katsumata et al. (2025)},
  year={2025},
  note={arXiv:2508.21102}
}

• arXiv: 2508.21102