Industrial Spill Detection Eval

industrial-spill-detection-eval

SynSpill: Improved Industrial Spill Detection With Synthetic Data — Baranwal et al. (2025) (arXiv:2508.10171, 2025)

What this evaluates

This benchmark evaluates an AI system's ability to detect and localize industrial safety hazards (e.g., oil spills, chemical stains) in complex factory environments. It probes the model's spatial grounding precision and its capacity to generalize from synthetic or limited real-world data to unseen operational sites.

Datasets

• Public Spill Data — total 1520; splits: test (520), train (100)
• Proprietary Factory Data — total 150; splits: icl (50), test (100)
• Synthetic Spill (SynSpill) Dataset — total 2000; splits: train (2000)

Metrics

• mean hit rate@IoU0.5 (primary) — range: [0, 1]
  • IoU = |B_pred ∩ B_gt| / |B_pred ∪ B_gt|. A predicted bounding box is counted as a hit if IoU ≥ 0.5. The metric reports the mean hit rate across all anomaly classes and test instances.

Input / output format

Input: RGB image paired with a structured prompt containing a system instruction ('You are a certified industrial safety inspector...') and a user instruction ('Detect and return the bounding-box coordinates of the in COCO JSON format, if present.').

Output: COCO JSON format containing bounding-box coordinates for detected anomaly classes.

Scoring recipe

hits = 0
total_gt = 0
for img in test_set:
    gt_boxes = img['gt_boxes']
    pred_boxes = parse_coco_json(model_output(img))
    total_gt += len(gt_boxes)
    for gt in gt_boxes:
        for pred in pred_boxes:
            if intersection_over_union(pred, gt) >= 0.5:
                hits += 1
                break
return hits / total_gt if total_gt > 0 else 0.0

Common pitfalls

• The evaluation uses a fixed IoU threshold of 0.5 rather than the standard COCO mAP protocol (IoU 0.5:0.95), which can inflate detection scores compared to standard benchmarks.
• Models must output strictly valid COCO JSON; minor formatting deviations (e.g., markdown fences, extra text) will cause parsing failures and zero scores.
• In-context learning (ICL) prepends 5, 10, or 15 examples, significantly altering context length and potentially affecting memory-bound VLMs differently than LoRA fine-tuning.

Evidence (verbatim from paper)

We report the mean hit rate at an IoU threshold of 0.5 across all anomaly classes. A predicted bbox is counted as a hit if it has sufficient overlap with the ground truth: IoU = |B_pred ∩ B_gt| / |B_pred ∪ B_gt|, Hit if IoU ≥ 0.5. This captures detection accuracy and spatial precision, aligning with requirements in safety-critical monitoring.

Citation

@misc{baranwal2025synspill,
  title={SynSpill: Improved Industrial Spill Detection With Synthetic Data},
  author={Baranwal et al. (2025)},
  year={2025},
  note={arXiv:2508.10171}
}

• arXiv: 2508.10171