Gmrpd Eval

gmrpd-eval

Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms — Wang et al. (2021) (arXiv:2103.02433, 2021)

What this evaluates

Evaluates a model's ability to perform pixel-level semantic segmentation for drivable areas and road anomalies using multi-modal visual inputs. It specifically probes how effectively networks can fuse RGB imagery with depth-related features (e.g., transformed disparity) to improve detection accuracy for ground mobile robots.

Datasets

• GMRP — total 3896; splits: train (2726), val (585), test (585)
• KITTI road — total 579; splits: train (289), test (290)
• KITTI semantic segmentation — total 400; splits: train (100), val (50), test (50)

Metrics

• IoU (primary) — range: [0, 1]
  • Intersection over Union: the ratio of the area of overlap between the predicted and ground truth masks to the area of their union. Computed per class and averaged for mIoU.
• F-score (Fsc) — range: [0, 1]
  • Harmonic mean of precision and recall at a given threshold. Computed per class and averaged for mFsc.
• Average Precision (AP) — range: [0, 1]
  • Area under the precision-recall curve for each class, computed by sweeping classification thresholds. Averaged across classes for mAP.

Input / output format

Input: Multi-modal image pairs (RGB, disparity, normal, elevation, HHA, or transformed disparity) downsampled to 320×480 resolution.

Output: Pixel-level segmentation masks indicating drivable areas and road anomalies.

Scoring recipe

def compute_iou(pred, gt):
    intersection = np.logical_and(pred, gt).sum()
    union = np.logical_or(pred, gt).sum()
    return intersection / union if union > 0 else 0.0

def compute_f1(pred, gt, threshold=0.5):
    pred_bin = pred > threshold
    tp = np.logical_and(pred_bin, gt).sum()
    fp = np.logical_and(pred_bin, ~gt).sum()
    fn = np.logical_and(~pred_bin, gt).sum()
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

def evaluate(pred_masks, gt_masks, num_classes):
    ious, fscs = [], []
    for c in range(num_classes):
        ious.append(compute_iou(pred_masks == c, gt_masks == c))
        fscs.append(compute_f1(pred_masks == c, gt_masks == c))
    return np.mean(ious), np.mean(fscs)

Common pitfalls

• KITTI road test set lacks ground truth, preventing direct metric computation; results must be submitted to an external benchmark server.
• KITTI semantic segmentation classes are merged into four new categories (unlabeled, drivable area, vehicles, pedestrians), altering the original evaluation schema.
• Input images are downsampled to 320×480, which may impact fine-grained anomaly detection compared to native resolutions.

Evidence (verbatim from paper)

For the quantitative evaluations, we adopt the F-score (Fsc) and the Intersection over Union (IoU) for each class. We also plot the precision-recall curves and compute the average precision (AP) for each class. Furthermore, we compute the mean values across all classes for the three metrics, denoted as mFsc, mIoU and mAP.

Citation

@misc{wang2021gmrpd,
  title={Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2103.02433}
}

• arXiv: 2103.02433