Cracknex Eval

cracknex-eval

CrackNex: a Few-shot Low-light Crack Segmentation Model Based on Retinex Theory for UAV Inspections — Yao et al. (2024) (arXiv:2403.03063, 2024)

What this evaluates

Evaluates few-shot crack segmentation performance under low-light conditions using illumination-invariant features. It tests the model's ability to generalize from well-illuminated support images to unseen low-light query images in both synthetic and real-world scenarios.

Datasets

• ll_CrackSeg9k — total 10500; splits: train (9000), test (1500)
• LCSD — total 143; splits: train (102), test (41)

Metrics

• mIOU (primary) — range: [0, 1]
  • Mean Intersection over Union: average IoU across all test images, where IoU = intersection of predicted and ground truth masks / union of predicted and ground truth masks.

Input / output format

Input: Support set of K well-illuminated crack images with binary masks, and a query image (synthetic or real low-light) to be segmented.

Output: Binary pixel-wise segmentation mask for the query image.

Scoring recipe

def compute_miou(preds, gts):
    ious = []
    for pred, gt in zip(preds, gts):
        intersection = np.logical_and(pred, gt).sum()
        union = np.logical_or(pred, gt).sum()
        if union == 0:
            ious.append(1.0)
        else:
            ious.append(intersection / union)
    return np.mean(ious)

Common pitfalls

• The ll_CrackSeg9k test set uses synthetic low-light images generated via Restormer, not real low-light photos, which may overestimate real-world performance.
• Few-shot evaluation requires reporting metrics separately for 1-shot and 5-shot settings; averaging them without distinction misrepresents performance.
• Training uses random horizontal flipping augmentation, but evaluation must be performed on original unaugmented images to match the reported protocol.

Evidence (verbatim from paper)

TABLE I: Baseline comparisons on the ll_CrackSeg9k and LCSD dataset in terms of mIOU↑ ... We select 9000 crack images from CrackSeg9k as our training set and another 1500 crack images as our test set. ... LCSD, with 102 well-illuminated crack images as the training set and 41 low-light crack images as the test set ... We further annotate each crack image pixel-wise and generate a binary label.

Citation

@misc{yao2024cracknex,
  title={CrackNex: a Few-shot Low-light Crack Segmentation Model Based on Retinex Theory for UAV Inspections},
  author={Yao et al. (2024)},
  year={2024},
  note={arXiv:2403.03063}
}

• arXiv: 2403.03063