Defect Spectrum Eval

defect-spectrum-eval

Defect Spectrum: A Granular Look of Large-Scale Defect Datasets with Rich Semantics — Yang et al. (2023) (arXiv:2310.17316, 2023)

What this evaluates

Evaluates industrial defect segmentation models by measuring their ability to accurately localize and classify multiple defect types within complex manufacturing images. It also assesses how well models trained on refined, granular annotations generalize compared to those trained on coarse original annotations, using both pixel-level and image-level quality control metrics.

Datasets

• Defect Spectrum — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union averaged across all defect classes. Computed as the average of IoU (intersection area divided by union area) for each class.
• Recall — range: [0, 1]
  • Image-level recall rate calculated as TP / (FN + TP), measuring the proportion of defective images correctly identified against expert-defined quality benchmarks.
• FPR — range: [0, 1]
  • False positive rate calculated as FP / (TN + FP), measuring the proportion of benign images incorrectly flagged as defective.

Input / output format

Input: RGB industrial defect images containing manufactured objects.

Output: Pixel-wise segmentation masks indicating defect boundaries and defect class labels per pixel.

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        ious.append(intersection / union if union > 0 else 1.0)
    return np.mean(ious)

def compute_recall_fpr(pred_labels, gt_labels):
    tp = np.sum((pred_labels == 1) & (gt_labels == 1))
    fn = np.sum((pred_labels == 0) & (gt_labels == 1))
    fp = np.sum((pred_labels == 1) & (gt_labels == 0))
    tn = np.sum((pred_labels == 0) & (gt_labels == 0))
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    fpr = fp / (tn + fp) if (tn + fp) > 0 else 0.0
    return recall, fpr

Common pitfalls

• Directly evaluating models trained on refined annotations against original coarse ground truth (or vice versa) is invalid due to annotation mismatch.
• Assuming a single architecture generalizes well across all defect categories; performance varies significantly by dataset (e.g., Transformers excel on Cotton-Fabric but struggle on others).
• Confusing defect detection or classification with the paper's explicit focus on fine-grained semantic segmentation.

Evidence (verbatim from paper)

For the performance metric, we choose the mean Intersection over Union (mIoU). Results are shown in Table 2. The consistent performance of DeepLabV3+ across multiple datasets suggests that it's a robust model for various types of defect segmentation tasks.

Citation

@misc{yang2023defectspectrum,
  title={Defect Spectrum: A Granular Look of Large-Scale Defect Datasets with Rich Semantics},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2310.17316}
}

• arXiv: 2310.17316