Pkugoodsad Eval

pkugoodsad-eval

PKU-GoodsAD: A Supermarket Goods Dataset for Unsupervised Anomaly Detection and Segmentation — Jian Zhang et al. (2023) (arXiv:2307.04956, 2023)

What this evaluates

Evaluates unsupervised visual anomaly detection and segmentation models on real-world supermarket goods. It probes robustness to object misalignment, intra-class appearance variation, and the ability to detect subtle or small anomalies without labeled anomalous training data.

Datasets

• PKU-GoodsAD — total 6124; splits: train (-1), test (-1); repo https://github.com/jianzhang96/GoodsAD

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across classification thresholds.
• AUPR (primary) — range: [0, 1]
  • Area Under the Precision-Recall curve. More sensitive to class imbalance than AUROC, commonly used for anomaly detection where anomalies are rare.
• PRO — range: [0, 1]
  • Per-Region Overlap. Measures the overlap between predicted and ground truth anomaly regions, normalized by region size, to balance performance across different anomaly sizes.

Input / output format

Input: RGB images of supermarket goods, resized to 224×224 pixels during training and testing. Ground truth pixel-level anomaly masks are provided for evaluation.

Output: Per-image anomaly score (for classification) and/or per-pixel anomaly score map (for segmentation/localization).

Scoring recipe

def compute_metrics(pred_scores, gt_labels, pred_masks=None, gt_masks=None):
    auroc = roc_auc_score(gt_labels, pred_scores)
    aupr = average_precision_score(gt_labels, pred_scores)
    pro = None
    if pred_masks is not None and gt_masks is not None:
        overlaps = []
        for p_mask, g_mask in zip(pred_masks, gt_masks):
            inter = np.sum(np.logical_and(p_mask, g_mask))
            union = np.sum(np.logical_or(p_mask, g_mask))
            if union > 0: overlaps.append(inter / union)
        pro = np.mean(overlaps) if overlaps else 0.0
    return {'AUROC': auroc, 'AUPR': aupr, 'PRO': pro}

Common pitfalls

• Object location misalignment across images breaks methods that assume fixed object positions.
• High intra-class appearance variation makes it difficult for reconstruction/distillation models to learn normal distributions.
• Pseudo-anomaly generation techniques (e.g., CutPaste, DRAEM) produce artifacts that differ significantly from natural retail anomalies, leading to poor generalization.

Evidence (verbatim from paper)

The standard classification metrics AUROC and AUPR are used for image-level anomaly classification and pixel-level anomaly segmentation. AUPR is more sensitive to the datasets of unbalanced categories. PRO [[50]] is also adopted to balance anomalous areas of different sizes.

Citation

@misc{zhang2023pkugoodsad,
  title={PKU-GoodsAD: A Supermarket Goods Dataset for Unsupervised Anomaly Detection and Segmentation},
  author={Jian Zhang et al. (2023)},
  year={2023},
  note={arXiv:2307.04956}
}

• arXiv: 2307.04956