Mmad Bbox Eval

mmad-bbox-eval

AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison — Jiang et al. (2026) (arXiv:2603.13779, 2026)

What this evaluates

Evaluates the fine-grained localization capability of vision-language models on industrial anomaly detection. It measures how accurately models can predict bounding boxes around defects compared to ground-truth annotations and human experts.

Datasets

• MMAD-BBox — total ?; splits: test (-1)

Metrics

• BBox-Mask IoU (primary) — range: [0, 1]
  • Converts predicted and ground-truth bounding boxes into binary masks and computes Intersection over Union (IoU) in the mask space. Additionally reports classification accuracy at IoU thresholds of 0.5, 0.3, 0.2, and 0.1.

Input / output format

Input: Industrial images paired with evaluation prompts requiring anomaly localization.

Output: Text-formatted bounding box coordinates predicting the anomaly region.

Scoring recipe

def compute_bbox_mask_iou(pred_boxes, gt_boxes):
    pred_masks = boxes_to_binary_masks(pred_boxes)
    gt_masks = boxes_to_binary_masks(gt_boxes)
    intersection = np.logical_and(pred_masks, gt_masks).sum(axis=(1,2))
    union = np.logical_or(pred_masks, gt_masks).sum(axis=(1,2))
    iou_scores = intersection / np.maximum(union, 1e-6)
    mean_iou = iou_scores.mean()
    accuracy_at_thresholds = {t: (iou_scores >= t).mean() for t in [0.5, 0.3, 0.2, 0.1]}
    return mean_iou, accuracy_at_thresholds

Common pitfalls

• Using conventional box-matching IoU instead of mask-space IoU, which fails for irregular or fragmented industrial defects.
• Applying standard object detection IoU thresholds (e.g., 0.5) without accounting for the inherent difficulty of anomaly localization, which requires lower thresholds (0.1-0.3) for meaningful evaluation.

Evidence (verbatim from paper)

Conventional box-matching IoU metrics can be unreliable, as industrial defects often have irregular shapes, disconnected fragments, or ambiguous instance boundaries. To provide a fairer and more consistent evaluation, we propose a new metric called BBox-Mask IoU. As illustrated in Fig.4, we convert both the predicted boxes and the ground-truth boxes into binary masks and compute IoU in the mask space. This alleviates bias caused by differing output granularities and offers a smooth, fine-grained measure of localization accuracy. In addition to the mean IoU, we report accuracy under IoU thresholds of 0.5, 0.3, 0.2, and 0.1, where relatively low thresholds are adopted as anomaly localization is inherently more challenging than standard object detection.

Citation

@misc{jiang2026adcopilot,
  title={AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2603.13779}
}

• arXiv: 2603.13779