Kolektor Sdd Eval

kolektor-sdd-eval

When Can We Trust Deep Neural Networks? Towards Reliable Industrial Deployment with an Interpretability Guide — Dong et al. (2026) (arXiv:2604.19206, 2026)

What this evaluates

Evaluates a training-free interpretability method (Δ-IoU) for detecting false negatives in binary industrial defect detection models. It probes whether post-hoc heatmap intersections can reliably flag 'in-distribution yet confidently wrong' predictions on surface defect datasets.

Datasets

• Kolektor SDD — total 399; splits: train (278), test (121)
• Kolektor SDD2 — total 3335; splits: train (2331), test (1004)

Metrics

• Recall (primary) — range: [0, 1]
  • Recall = TP / (TP + FN). Measures the proportion of actual defective samples correctly identified by the model or the suspicious sample detection method.
• Accuracy — range: [0, 1]
  • Accuracy = (TP + TN) / (TP + TN + FP + FN). Measures the overall proportion of correctly classified samples.

Input / output format

Input: Surface defect images of electronic commutators labeled as defective or defect-free.

Output: Binary classification label and a Δ-IoU score for suspicious sample detection (threshold β=0.2).

Scoring recipe

def compute_metrics(TP, TN, FP, FN):
    recall = TP / (TP + FN) if (TP + FN) > 0 else 0.0
    accuracy = (TP + TN) / (TP + TN + FP + FN)
    return recall, accuracy

Common pitfalls

• The Δ-IoU threshold β=0.2 is used for suspicious detection but is noted to be adjustable; changing it alters the recall-accuracy trade-off.
• Adversarial enhancement achieves 100% recall but drastically increases false positives (misclassified true negatives), which must be accounted for when evaluating safety-critical deployment.
• Confidence-based thresholding is used as a baseline but fundamentally fails to detect model errors, making it an inappropriate comparison for error detection.

Evidence (verbatim from paper)

We employ several commonly used metrics in classification problems to comprehensively evaluate the performance of the suspicious sample detection task, specifically the counts of true negatives (TN) and false negatives (FN), as well as recall and accuracy. The formula for recall is given as follows: Recall = TP / (TP + FN).

Citation

@misc{dong2026trust,
  title={When Can We Trust Deep Neural Networks? Towards Reliable Industrial Deployment with an Interpretability Guide},
  author={Dong et al. (2026)},
  year={2026},
  note={arXiv:2604.19206}
}

• arXiv: 2604.19206