Fishyscapes Eval

fishyscapes-eval

Pixel-wise Anomaly Detection in Complex Driving Scenes — Di Biase et al. (2021) (arXiv:2103.05445, 2021)

What this evaluates

This benchmark probes a model's ability to perform pixel-wise anomaly detection and uncertainty estimation in complex urban driving scenes. It specifically measures how well a segmentation wrapper identifies out-of-distribution objects (e.g., lost & found items, static blends, web overlays) without degrading the underlying semantic segmentation accuracy.

Datasets

• Fishyscapes benchmark — total ?; splits: FS Lost & Found (275), FS Static (1000), FS Web Oct. 2020 (-1)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision: the area under the precision-recall curve computed over all pixel-wise anomaly scores.
• FPR95 — range: percent
  • False Positive Rate at 95% True Positive Rate: the false positive rate on the ROC curve when the true positive rate reaches 95%.

Input / output format

Input: Raw driving scene images (Cityscapes resolution 2048x1024) processed by a pre-trained semantic segmentation network.

Output: Pixel-wise anomaly/uncertainty scores or maps indicating the likelihood of out-of-distribution objects.

Scoring recipe

# predictions: flattened array of pixel-wise anomaly scores
# gold: flattened array of binary anomaly masks (1=anomaly, 0=background)
fpr, tpr, _ = roc_curve(gold, predictions)
ap = average_precision_score(gold, predictions)
# Find FPR at TPR >= 0.95
fpr95 = fpr[tpr >= 0.95][0]
return {"AP": ap, "FPR95": fpr95}

Common pitfalls

• Using ROC AUC instead of AP/FPR95, which the authors note is ill-suited for highly imbalanced anomaly detection.
• Failing to account for segmentation accuracy trade-offs; methods that retrain the backbone to detect anomalies often compromise mIOU, so wrapper methods must be compared against non-retrained baselines.
• Evaluating on the wrong FS Web crawl version, as the dataset dynamically updates and results are only comparable if matched to the same submission date (e.g., Oct 2020).

Evidence (verbatim from paper)

To assess the performance of the framework against existing methods, we use the same metrics presented in the Fishyscapes benchmark for anomaly detection: average precision (AP) and the false positive rate at 95% true positive rate (FPR95).

Citation

@misc{dibiase2021pixelwise,
  title={Pixel-wise Anomaly Detection in Complex Driving Scenes},
  author={Di Biase et al. (2021)},
  year={2021},
  note={arXiv:2103.05445}
}

• arXiv: 2103.05445