Segmentmeifyoucan Eval

segmentmeifyoucan-eval

SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation — Chan et al. (2021) (arXiv:2104.14812, 2021)

What this evaluates

This benchmark evaluates a model's ability to segment anomalous or hazardous objects in driving scenes that were not seen during training. It probes out-of-distribution detection and pixel-wise localization of unknown road obstacles, emphasizing safety-critical detection regardless of object class.

Datasets

• RoadAnomaly21 — total 100; splits: test (100); repo https://github.com/SegmentMeIfYouCan/road-anomaly-benchmark
• RoadObstacle21 — total 327; splits: test (327); repo https://github.com/SegmentMeIfYouCan/road-anomaly-benchmark

Metrics

• AuPRC (primary) — range: [0, 1]
  • Area under the precision-recall curve computed over pixel-wise anomaly scores. Emphasizes detecting the minority anomaly class in highly unbalanced pixel distributions.
• FPR95 — range: percent
  • False positive rate when the true positive rate reaches 95%. Indicates how many false positives must be made to achieve the desired safety threshold.
• F1 — range: [0, 1]
  • Component-wise F1-score: F1(τ) = 2·TP(τ) / (2·TP(τ) + FN(τ) + FP(τ)). Summarizes true positives, false negatives, and false positives at a given threshold τ.

Input / output format

Input: RGB image of a driving scene. The model receives the image and outputs pixel-wise anomaly scores or a binary segmentation mask.

Output: Pixel-wise anomaly scores s(x) ∈ ℝ^|Z| or a binary mask. Predictions are thresholded at τ ∈ [0,1) to form connected components for component-level evaluation.

Scoring recipe

def compute_metrics(gold_mask, pred_scores, tau=0.5):
    # Pixel-level
    au_prc = auc_precision_recall(gold_mask, pred_scores)
    fpr95 = fpr_at_tpr(gold_mask, pred_scores, target_tpr=0.95)

    # Component-level
    gt_comps = find_connected_components(gold_mask, class='anomaly')
    pred_comps = find_connected_components(pred_scores > tau, class='anomaly')
    tp, fn, fp = 0, 0, 0
    for k in gt_comps:
        union_k = union(p for p in pred_comps if p & k)
        sIoU_k = len(k & union_k) / len((k | union_k) - A(k))
        tp += 1 if sIoU_k > tau else 0
        fn += 1 if sIoU_k <= tau else 0
    for k_hat in pred_comps:
        ppv_k = len(k_hat & union(g for g in gt_comps if g & k_hat)) / len(k_hat)
        fp += 1 if ppv_k <= tau else 0
    f1 = 2 * tp / (2 * tp + fn + fp)
    return au_prc, fpr95, f1

Common pitfalls

• Void regions must be strictly excluded from evaluation; predictions falling in void areas are not counted as false positives.
• Component-level metrics use adjusted IoU (sIoU) that excludes pixels correctly intersecting other ground-truth components, unlike standard IoU which penalizes covering multiple objects.
• Threshold τ significantly impacts component-level TP/FN/FP counts; results must be reported at specific thresholds (0.25, 0.50, 0.75) or averaged, not assumed.

Evidence (verbatim from paper)

We evaluate the separability of the pixel-wise anomaly scores via the area under the precision-recall curve (AuPRC)... To consider the safety point of view, we also include the false positive rate at 95% true positive rate (FPR95) in our evaluation... As an overall metric, we additionally include the component-wise F1-score defined as F1(τ) = 2·TP(τ) / (2·TP(τ) + FN(τ) + FP(τ)) ∈ [0,1], which summarizes the TP, FN and FP quantities (that depend on τ).

Citation

@misc{chan2021segmentmeifyoucan,
  title={SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation},
  author={Chan et al. (2021)},
  year={2021},
  note={arXiv:2104.14812}
}

• arXiv: 2104.14812