Dacl10k Eval

dacl10k-eval

dacl10k: Benchmark for Semantic Bridge Damage Segmentation — Flotzinger et al. (2023) (arXiv:2309.00460, 2023)

What this evaluates

Evaluates the ability of computer vision models to perform multi-label semantic segmentation for identifying and localizing various reinforced concrete defects on bridge infrastructure. It probes pixel-level classification accuracy across diverse, real-world damage types and structural components.

Datasets

• dacl10k — total 9920; splits: test (-1)

Metrics

• mean IoU (primary) — range: [0, 1]
  • Mean Intersection over Union calculated across all damage classes. For each class, IoU is the intersection of predicted and ground truth masks divided by their union, averaged over the dataset.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall computed per class and averaged. Precision is the ratio of true positive pixels to all predicted positive pixels, while recall is the ratio of true positive pixels to all actual positive pixels.

Input / output format

Input: RGB images of bridge surfaces captured via UAV or smartphone, containing one or more concrete defects.

Output: Multi-label pixel-wise segmentation masks assigning each pixel to one of the defined damage classes or background.

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        if union > 0:
            ious.append(intersection / union)
        else:
            ious.append(0.0)
    return np.mean(ious)

Common pitfalls

• Hollowareas are not visually recognizable defects but are annotated as yellow/red/blue chalk markings made during hands-on inspections, requiring models to learn annotation artifacts rather than physical damage.
• Significant class imbalance exists, with some classes like WConcor having only 33 images while others like Spalling have over 1000, skewing average metrics if not handled properly.
• Multi-label nature means multiple defects can overlap or co-occur in a single image, complicating pixel-wise boundary delineation.

Evidence (verbatim from paper)

Baseline models using deep semantic segmentation architectures achieve a mean IoU of 0.42 on the test set, demonstrating the dataset's utility for evaluating automated damage recognition in civil infrastructure. In Table 6 we display additional metrics from the auxiliary head of the best model on the test split.

Citation

@misc{flotzinger2023dacl10k,
  title={dacl10k: Benchmark for Semantic Bridge Damage Segmentation},
  author={Flotzinger et al. (2023)},
  year={2023},
  note={arXiv:2309.00460}
}

• arXiv: 2309.00460